Add ability to limit tests by test duration rather than the number of test iterations

[vlsi]

Testing Problem

I'm testing database-related integration cases like "create table", "drop table", and the sequence of operations is generated on the fly (see #134), so @Property(tries=42) is not suitable for me as I have no idea how much time a single iteration would take.

Suggested Solution

Support something like @Timeout from JUnit5: https://junit.org/junit5/docs/5.5.1/api/org/junit/jupiter/api/Timeout.html

The "test timeout" feature could be implemented via AfterProperty + AroundTryHook, however, having an out-of-the-box timeout would be perfect.

See https://www.fuzzingbook.org/html/WhenToStopFuzzing.html, https://docs.gitlab.com/ee/user/application_security/coverage_fuzzing/#duration-of-coverage-guided-fuzz-testing

[jlink]

Should the time limit be in addition to number of tries or replace the number of tries?
What I mean @Property(tries = 10000, maxSeconds= 10) would stop depending on which occurs first: 10000 tries or 10 seconds.

[vlsi]

@Property(tries = 10000, maxSeconds= 10)

That makes sense, and I have pretty much the same intention: allow stoping the test as soon as the first limit is exhausted.

As I started adding my own @Timeout I immediately realized I want to be able to pass the timeout from the command line to allow different CI jobs to have different fuzz durations.

I'm not sure what to do about it, and it might be that a generic "pass values from the command line" (e.g. via system properties) might be helpful.

For instance, what if jqwik picks up jqwilk.proprerty.com.acme.MyProperties.propertyName.tries=42 and uses that as an override for @Property on top of propertyName method in com.acme.MyProperties class?

[jlink]

For instance, what if jqwik picks up jqwilk.proprerty.com.acme.MyProperties.propertyName.tries=42

This would make the most sense as a generic feature for all attributes accepted by @Property.

[vlsi]

This would make the most sense as a generic feature for all attributes accepted by @Property

I moved the discussion to #337

In any case, I think it would be fine to put timeout to @Property(...) (and @PropertyDefaults).

[vlsi]

Well, it is awesome there's net.jqwik.api.lifecycle.TryExecutionResult#satisfied(shouldPropertyFinishEarly=true) which would do exactly what I need for the timeout feature :)

[jlink]

This will be supported by jqwik 2. So I close the feature request for jqwik 1.

[vlsi]

Is there an issue tracker for jqwik 2?

[jlink]

Not yet, but there will be.