direct parsing fuzzing test

[tyler92]

Two issues were found by the new fuzzing target: #1047 and #1048

I think detecting memory issues in case of new changes or refactoring might be useful.
I'm not sure about code formatting for the project, I will appreciate your feedback.

[cppalliance-bot]

An automated preview of the documentation is available at https://1052.jsondocs.prtest.cppalliance.org/libs/json/doc/html/index.html

[cppalliance-bot]

An automated preview of the documentation is available at https://1052.jsondocs.prtest.cppalliance.org/libs/json/doc/html/index.html

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 93.41%. Comparing base (502ac79) to head (3f88a33).
Report is 2 commits behind head on develop.

Additional details and impacted files

@@           Coverage Diff            @@
##           develop    #1052   +/-   ##
========================================
  Coverage    93.41%   93.41%           
========================================
  Files           91       91           
  Lines         8667     8667           
========================================
  Hits          8096     8096           
  Misses         571      571           

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 502ac79...3f88a33. Read the comment docs.

[cppalliance-bot]

Benchmark test results. More info at https://benchmark.cppalliance.org/jsonbenchmarks-pullrequests/1052/pullrequest.html

[cppalliance-bot]

An automated preview of the documentation is available at https://1052.jsondocs.prtest.cppalliance.org/libs/json/doc/html/index.html

[cppalliance-bot]

Benchmark test results. More info at https://benchmark.cppalliance.org/jsonbenchmarks-pullrequests/1052/pullrequest.html

[grisumbras]

While the idea is interesting, I am sceptical fuzzing with random strings can be effective when the parser expects a very specific structure of the input. It will probably always fail on the second (on_object_begin), the third (on_key), or the fourth (on_bool) event.

[tyler92]

According to coverage reports, it works well enough to find valid inputs, these two issues were found quite quickly. We can provide a dictionary for the fuzzer to help it a bit if we want to improve the performance.

[sdarwin]

It may be a coincidence. This pull request mentions "detecting memory issues". The drone agent on IBM s390 crashed with out-of-memory errors. I have rebooted the server.

[grisumbras]

It's also possible that linking the new fuzzer requires too much memory (variant parsers do do that).

[tyler92]

I am sceptical fuzzing with random strings can be effective when the parser expects a very specific structure of the input. It will probably always fail on the second (on_object_begin), the third (on_key), or the fourth (on_bool) event.

Just for an experiment: I reverted the fix for #1047 and measured the time required to find that bug. Without a dictionary and with an empty corpus (the worst scenario) it takes about 10 seconds. The reason is the fuzzer doesn't have to find a valid input with all required fields. It's enough to generate a JSON with some fields to verify parser for a specific type only.

Adding corpus doesn't improve speed much. But I've added two valid JSONs to seed files, and it gives the fuzzer the ability to reach the "success" point. With this change, the fuzzer finds #1047 in less than one second. Also, I changed C++ version to C++14 to avoid potential OOM.

Please let me know if it makes sense

[cppalliance-bot]

An automated preview of the documentation is available at https://1052.jsondocs.prtest.cppalliance.org/libs/json/doc/html/index.html

[cppalliance-bot]

Benchmark test results. More info at https://benchmark.cppalliance.org/jsonbenchmarks-pullrequests/1052/pullrequest.html

[grisumbras]

Please, replace with boost::variant2::variant. It will reduce the amount of macro switching necessary.

[tyler92]

Done

[grisumbras]

Why have you reduced this from 17 to 14?

[tyler92]

I wanted to make sure this MR has a chance to be approved and avoid potential OOMs on the drone agent before (although I'm not sure if it's related to the current MR). I reverted to 17; please let me know if there are any issues with it.

[cppalliance-bot]

An automated preview of the documentation is available at https://1052.jsondocs.prtest.cppalliance.org/libs/json/doc/html/index.html

[cppalliance-bot]

Benchmark test results. More info at https://benchmark.cppalliance.org/jsonbenchmarks-pullrequests/1052/pullrequest.html

[grisumbras]

Everything looks good. Can you please squsah and rebase on the current develop (if it's not already). I can also do it manually myself, but then the PR won't be considered merged by GitHub.

[cppalliance-bot]

An automated preview of the documentation is available at https://1052.jsondocs.prtest.cppalliance.org/libs/json/doc/html/index.html

[cppalliance-bot]

An automated preview of the documentation is available at https://1052.jsondocs.prtest.cppalliance.org/libs/json/doc/html/index.html

[tyler92]

Can you please squsah and rebase on the current develop (if it's not already). I can also do it manually myself, but then the PR won't be considered merged by GitHub.

Done

[cppalliance-bot]

Benchmark test results. More info at https://benchmark.cppalliance.org/jsonbenchmarks-pullrequests/1052/pullrequest.html

[grisumbras]

Thank you for your contribution.