Retry hurl files as whole

[simonzkl]

Problem to solve

When running in test mode, I'd like to be able to retry Hurl files as a whole, as opposed to retrying individual requests. The reason I'd like to add retries is to catch flaky tests (a test fails, but passes after a subsequent retry).

The per-request retry mechanism is not ideal for catching flaky tests:

1. Per-request retry is often used as a workaround for eventually consistent APIs that take some time to refresh. Hence they're expected temporary failures, not indicitative of a flaky test.
2. Some requests might not be idempotent, and cannot be retried without starting from the beginning of the test.

Proposal

• Add a flag (e.g. --test-retry <n>) which retries the entire test file from the beginning
• Should the test suite as a whole pass if all tests pass after retries?
  • For detecting flaky tests, you might still want to exit with a non-zero exit code.
  • Possibly configurable through another flag?
• JUnit reports should log all test attempts, not just successful ones so it's easy to spot flaky tests from reports.

Additional context and resources

N/A

Tasks to complete

TODO

[jcamiel]

Hi @simonzkl

This could be potentially difficult to implement: for instance, in --test mode, tests are run in parallel and the list of all tests to execute is constructed at the start of the run. Adding test files "on the fly" because they've failed is feasible but can be tricky to implement with the current code.

For your use case, it is possible to use --repeat option? If you have 3 files for instance:

$ hurl --test --repeat 10 a.hurl b.hurl c.hurl

will run (a, b, c) in loop 10 times.

I find it even better than a flag that retry only failed test because if a test is flaky, it can run OK the first time and not being identified as flaky (and the option --repeat already exists of course!)

[simonzkl]

Hm, the test suite can actually take a while to run, so the preference would be to only retry those that failed. I'm not trying to explicitly look for flaky tests. Just mark tests as flaky as the failures occur in regular CI pipelines without significantly extending the runtime of those pipelines. Running tests in repeat would be more fitting on a schedule.

If it's tricky to retry tests on the spot, maybe it would be easier to aggregate them and re-run them at the end? I suppose I could also write a wrapper script to do this if the use case seems niche. Though I feel like this is pretty common practice in test runners.