Running fuzzers

[bradh]

As part of the opaque properties writing, I need to address a requirement for fuzzing of the API (see
#2510 (comment) )

I'm struggling to understand how the fuzzers are supposed to be run. If I turn on the likely-looking cmake options (AVIF_ENABLE_FUZZTEST and AVIF_LOCAL_FUZZTEST) then it looks like fuzzers build OK, and make test does appear to be running fuzzers, but some of them fail:

      Start 44: avif_fuzztest_dec
44/58 Test #44: avif_fuzztest_dec ................   Passed    1.09 sec
      Start 45: avif_fuzztest_dec_incr
45/58 Test #45: avif_fuzztest_dec_incr ...........   Passed    1.08 sec
      Start 46: avif_fuzztest_enc_dec
46/58 Test #46: avif_fuzztest_enc_dec ............***Failed    0.76 sec
      Start 47: avif_fuzztest_enc_dec_anim
47/58 Test #47: avif_fuzztest_enc_dec_anim .......***Failed    0.13 sec
      Start 48: avif_fuzztest_enc_dec_incr
48/58 Test #48: avif_fuzztest_enc_dec_incr .......   Passed    1.05 sec
      Start 49: avif_fuzztest_read_image
49/58 Test #49: avif_fuzztest_read_image .........   Passed    1.10 sec
      Start 50: avif_fuzztest_yuvrgb
50/58 Test #50: avif_fuzztest_yuvrgb .............***Failed    0.02 sec

In addition, those passes seem to be pretty short duration - I second. So I think I'm doing it wrong.

Is there a recipe for making the fuzzers pass and do meaningful fuzzing?

[vrabaud]

Hi, I just added instructions for fuzzers here: #2523 That's only from a docker.

For the failing tests, what was your exact CMake command ? I will update the doc for that too. You need local libaom, dav1d, libyuv and libsharpyuv.

[bradh]

I used ccmake, so I can't tell you an equivalent command. I can say that I did have AOM, and I think I had libyuv. I didn't have dav1d, and almost certainly didn't have libsharpyuv.

I have rebuild it (not docker) with more deps and successfully run it from make test.

[maryla-uc]

Hi, sorry for the lack of instructions on how to run the fuzzers.
In general, to see why the test fails you should run it with logs, the easiest being with a command like this:

ctest --output-on-failure -R fuzztest

(the value after -R is a regular expression for the tests to run)

avif_fuzztest_yuvrgb probably fails because you don't have AVIF_LIBSHARPYUV enabled (a requirement which again, was not properly documented).

Update: Another requirement is having both AOM and DAV1D enabled. tests/CMakeLists.txt lists the following recommended flags (but I think some are missing): -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ -DAVIF_CODEC_DAV1D=ON -DAVIF_ENABLE_WERROR=OFF

If you have all the right flags, then the failures might be related to your change.

The logs should show you a reproducer test that you can copy paste in the test file to rerun it easily, OR you can use the FUZZTEST_PRNG_SEED which is printed as well, e.g.:

FUZZTEST_PRNG_SEED=wDH4mhJJCJa1aCoRe-yN-Jlzqn7oa9RxFnltRR1y1_A ctest --output-on-failure -R ^avif_fuzztest_enc_dec$

It is expected that the tests run only for a second or so, because they are in "unit test" mode, see also the fuzztest documentation. I have not tried running them in "fuzzing mode" mode locally, but you could also simply rerun them several times if you want to fuzz more.
We do also have continuous fuzzing infrastructure which runs the tests for longer once they're submitted.

[bradh]

As noted above, it looks like dav1d was the missing piece. Its now running for me (from master branch).