Soft fail tests

[jeff-cook]

Has anyone found a way to soft fail a test?

Use cases...

• new tests that might not be stable yet, but informative enough to share the information.
• some times I find I want to get info in front of the users and the junit reports in my CI only shows failed tests.

My current work around, is to run bats filtering for soft fail tests and allow the bats command to fail. Then run bats again filtering out soft fail test tag. Because I often run tests in parallel it would be great to be able to run all of this as a single bats command.

Thanks

[martin-schulze-vireso]

Whit exactly do you mean with softfail? Especially, what should the return code of bats be if a softfail occurs?

[jeff-cook]

In this case some fail (for some tests, but not all)

• Have bats mark the test as failed for reporting.
  • Including on stdout and in reports such a junit
• Have the bats command exit 0 if only tests tagged as softfail:true fail.
  • Exit 1 if other tests fail.