Is there a way to list project test suites?

[philderbeast]

Is there a way to list project test suites?

I'd like to write all of a project's tests (in target form) to a file, discard the long-running ones, then have cabal run only the remaining tests.

I'm able to use cabal test all --dry-run but this noisy. Ideally I'd like a fully qualified target form like package:test:name:

$ cabal test all --dry-run
Resolving dependencies...
Build profile: -w ghc-9.4.5 -O1
In order, the following would be built (use -v for more details):
 - Cabal-syntax-3.9.0.0 (lib) (first run)
 - HTTP-4000.4.1 (lib) (requires build)
 - cryptohash-sha256-0.11.102.1 (lib) (requires build)
 - distributive-0.6.2.1 (lib) (requires build)
 - echo-0.1.4 (lib) (requires build)
 - edit-distance-0.2.2.1 (lib) (requires build)
 - lukko-0.1.1.3 (lib) (requires build)
 - optparse-applicative-0.16.1.0 (lib) (requires build)
 - rere-0.2 (lib) (requires build)
 - resolv-0.1.2.0 (lib:resolv) (requires build)
 - these-1.1.1.1 (lib) (requires build)
 - vector-0.13.0.0 (lib) (requires build)
 - Cabal-3.9.0.0 (lib) (first run)
 - comonad-5.0.8 (lib) (requires build)
 - hackage-security-0.6.2.3 (lib) (first run)
 - tasty-1.4.3 (lib) (requires build)
 - vector-th-unbox-0.2.2 (lib) (requires build)
 - vector-binary-instances-0.2.5.2 (lib) (requires build)
 - nothunks-0.1.4 (lib) (requires download & build)
 - math-functions-0.3.4.2 (lib) (requires build)
 - indexed-traversable-instances-0.1.1.2 (lib) (requires build)
 - cassava-0.5.3.0 (lib) (requires build)
 - bitvec-1.1.4.0 (lib) (requires build)
 - cabal-install-solver-3.9.0.0 (lib) (first run)
 - Cabal-tests-3 (test:custom-setup-tests) (first run)
 - Cabal-QuickCheck-3.9.0.0 (lib) (first run)
 - bifunctors-5.5.15 (lib) (requires build)
 - tasty-quickcheck-0.10.2 (lib) (requires build)
 - tasty-hunit-0.10.0.3 (lib) (requires build)
 - tasty-golden-2.3.5 (lib) (requires build)
 - tasty-expected-failure-0.12.3 (lib) (requires build)
 - mwc-random-0.15.0.2 (lib) (requires build)
 - witherable-0.4.2 (lib) (requires build)
 - vector-algorithms-0.9.0.1 (lib) (requires build)
 - cabal-install-3.9.0.0 (lib) (first run)
 - semigroupoids-5.3.7 (lib) (requires build)
 - assoc-1.0.2 (lib) (requires build)
 - Cabal-described-3.9.0.0 (lib) (first run)
 - cabal-install-solver-3.9.0.0 (test:unit-tests) (first run)
 - Cabal-tests-3 (test:rpmvercmp) (first run)
 - Cabal-tests-3 (test:no-thunks-test) (first run)
 - Cabal-tests-3 (test:check-tests) (first run)
 - dense-linear-algebra-0.1.0.0 (lib) (requires build)
 - cabal-install-3.9.0.0 (test:mem-use-tests) (first run)
 - cabal-install-3.9.0.0 (test:integration-tests2) (first run)
 - semialign-1.2.0.1 (lib) (requires build)
 - strict-0.4.0.1 (lib) (requires build)
 - cabal-install-3.9.0.0 (test:long-tests) (first run)
 - Cabal-tests-3 (test:unit-tests) (first run)
 - aeson-2.1.2.1 (lib) (requires build)
 - tree-diff-0.2.2 (lib) (requires build)
 - statistics-0.16.2.0 (lib) (requires build)
 - microstache-1.0.2.3 (lib) (requires build)
 - criterion-measurement-0.1.4.0 (lib) (requires build)
 - Cabal-tree-diff-3.9.0.0 (lib) (first run)
 - solver-benchmarks-3 (lib) (first run)
 - criterion-1.5.13.0 (lib) (requires build)
 - cabal-install-3.9.0.0 (test:unit-tests) (first run)
 - Cabal-tests-3 (test:parser-tests) (first run)
 - Cabal-tests-3 (test:hackage-tests) (first run)
 - solver-benchmarks-3 (test:unit-tests) (first run)
 - cabal-benchmarks-3 (test:cabal-benchmarks) (first run)

[ulysses4ever]

This is much needed: I had a question like this just this week.

Option 1 (cabal list-bin):

❯ cabal list-bin all
Error: cabal: The list-bin command is for finding a single binary at once. The
target 'all' refers to all the packages in the project which includes the test
suite 'mypkg-test-2' and the test suite 'mypkg-test'.

You'd need to grep through this, perhaps. There's a feature request to make it more structured (#8189 (comment)).

Option 2 (cabal-plan):

❯ cabal-plan list-bins
mypkg:test:mypkg-test  /home/artem/tmp/mypkg/dist-newstyle/build/x86_64-linux/ghc-9.2.7/mypkg-0.1.0.0/t/mypkg-test/build/mypkg-test/mypkg-test
mypkg:test:mypkg-test-2  /home/artem/tmp/mypkg/dist-newstyle/build/x86_64-linux/ghc-9.2.7/mypkg-0.1.0.0/t/mypkg-test-2/build/mypkg-test-2/mypkg-test-2

[philderbeast]

Thanks @ulysses4ever, the cabal-plan output shows the most promise for unmangling.

[philderbeast]

This is what I did to get the list of test targets cabal-plan list-bins > plan.txt and then with vim editing that file; :%s/ \//.*$// followed by :g/.*:exe:.*/d followed by :g/.*:bench:.*/d.

To run those test targets I used cabal test $(cat plan.txt).

[ulysses4ever]

:%s/ \//.*$//

I think it's supposed to be :%s/ \/.*$// (extra forward slash in the middle). A bash one-liner that should mimick your vim commands:

> cat plan.txt | cut -d' ' -f1 | sed '/:exe:/d' | sed '/:bench:/d'

(provided you don't have anything funny in your paths that sed would choke on.)

[philderbeast]

Thanks @ulysses4ever for the one-liner. Yes I made a transcription error with the :%s... regex substitution.

[fgaz]

This is much needed

agreed! we have to hack around the lack of a list-targets command even in this repo. for example here's a cursed oneliner that could be removed:

cabal/cabal-install/bash-completion/cabal

Line 111 in 9bf415a

grep -E -i "^[[:space:]]*($CabalComponent)[[:space:]]" "$FILE" 2>/dev/null | sed -e "s/.* \([^ ]*\).*/\1/" | sed -e '/^$/d'

[mouse07410]

I'd like to resurrect the cabal list-bin all.

I think it's absolutely vital fire normal users to get it work as expected - i.e., listing all of the binaries generated for this project.

I admit I can't appreciate how difficult or easy it would be to get this capability working.

[ulysses4ever]

@mouse07410 sure. I'd take inspiration from above mentioned cabal-plan.

[philderbeast]

I'd like to resurrect the cabal list-bin all.

Would that give us something like cabal list-bin all:exes and cabal list-bin all:tests?

[philderbeast]

@ulysses4ever, @mouse07410 and @fgaz are you available to review #9744 that fixes this?

[ulysses4ever]

@philderbeast I brought it up on the last Cabal meeting. I'm personally happy with what I see so far, but I don't understand how design considerations were solved; e.g. separate subcommand vs. a flag for cabal build. I want to read the whole thread before chiming in.