-
Notifications
You must be signed in to change notification settings - Fork 3
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
1 parent
33a9b10
commit 5af7ba0
Showing
60 changed files
with
8,364 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,4 @@ | ||
# Sphinx build info version 1 | ||
# This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done. | ||
config: c1fffd718f2a7b6b1b87397abc933d3b | ||
tags: 645f666f9bcd5a90fca523b33c5a78b7 |
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,74 @@ | ||
# Changelog | ||
HypoFuzz uses [calendar-based versioning](https://calver.org/), with a | ||
`YY-MM-patch` format. | ||
|
||
## 24.02.3 | ||
Fixed compatibility with Pytest 8.x ([#32](https://github.com/Zac-HD/hypofuzz/issues/32)). | ||
|
||
## 24.02.2 | ||
Fixed a dashboard bug ([#31](https://github.com/Zac-HD/hypofuzz/issues/31)). | ||
|
||
## 24.02.1 | ||
Now requires [Hypothesis 6.93.2](https://hypothesis.readthedocs.io/en/latest/changes.html#v6-93-2) | ||
or later, fixing compatibility with some unstable internals that HypoFuzz hooks into (yes, again). | ||
Also deduplicates the displayed covering examples in the dashboard, when their reprs are identical. | ||
|
||
## 23.12.1 | ||
Now requires [Hypothesis 6.91](https://hypothesis.readthedocs.io/en/latest/changes.html#v6-91-0) | ||
or later, fixing compatibility with some unstable internals that HypoFuzz hooks into. | ||
|
||
## 23.07.1 | ||
Various small patches for issues found by fuzzing Hypothesis itself. | ||
Notably, we now try to fuzz functions which require autouse fixtures | ||
even though we don't provide those fixtures - it often works! | ||
|
||
## 23.06.1 | ||
Hypofuzz now writes ``.patch`` files with failing examples, *and* optionally | ||
with covering examples - including removal of redundant covering examples. | ||
|
||
## 23.05.3 | ||
Yet another compatibility fix for explain mode, with even _more_ tests for future regressions. | ||
Third time lucky, we hope! | ||
|
||
## 23.05.2 | ||
Additional compatibility fix, and improved tests to avoid future regressions. | ||
|
||
## 23.05.1 | ||
Now requires [Hypothesis 6.75.2](https://hypothesis.readthedocs.io/en/latest/changes.html#v6-75-2) | ||
or later, fixing compatibility with some unstable internals that HypoFuzz hooks into. | ||
|
||
## 23.04.1 | ||
Fixed a bug affecting traceback reporting on Python 3.10+ (#16). | ||
|
||
## 22.12.1 | ||
Fixed a `NotImplementedError` when HypoFuzz had found a failing input for *every* | ||
test running in some process. Now, we cleanly `sys.exit(1)` instead. | ||
|
||
## 22.10.1 | ||
First open-source release! Also improves database handling and adds | ||
a timelimit to shrinking to better handle pathological cases. | ||
|
||
## 22.07.1 | ||
Fixes compatibility with [Hypothesis 6.49.1](https://hypothesis.readthedocs.io/en/latest/changes.html#v6-49-1) | ||
and later (released July 5th). | ||
|
||
## 21.12.2 | ||
Web-based time-travel debugging with [PyTrace](https://pytrace.com/) - just | ||
`pip install hypofuzz[pytrace]` and click the link on any failing test! | ||
|
||
## 21.12.1 | ||
Improved dashboard (again!), better generation/mutation heuristics, | ||
basic residual-risk estimation, free-for-noncommercial-use on PyPI. | ||
|
||
## 21.05.1 | ||
Improved dashboard, fixed memory leaks, support arbitary `hypothesis.event()` | ||
calls as coverage, performance improvements. | ||
|
||
## 20.09.1 | ||
Multiprocess fuzzing with seed sharing and corpus distillation, | ||
and basic prioritisation of seeds covering rare branches. | ||
|
||
Still very much in alpha, but the fundamentals all work now. | ||
|
||
## 20.08.1 | ||
First packaged version of HypoFuzz |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,61 @@ | ||
Configuration | ||
============= | ||
|
||
.. contents:: | ||
:local: | ||
|
||
|
||
It's all automatic | ||
------------------ | ||
|
||
Wherever possible, HypoFuzz is designed to do the right thing without configuration. | ||
|
||
The :command:`hypothesis fuzz` CLI accepts arguments to determine parameters like the | ||
number of processes to use, and the port on which to serve the dashboard - | ||
though every one of them is optional - and you can select which tests to fuzz | ||
in the same way you would select them via :pypi:`pytest`. | ||
|
||
Once that's all set up though, the fuzzer is not configurable: manual prioritization | ||
of a fuzzing process is considerably more error-prone than allowing the adaptive | ||
scheduler to do its thing, and computer time is much cheaper than yours. | ||
|
||
|
||
|
||
Custom coverage events | ||
---------------------- | ||
|
||
HypoFuzz runs :pypi:`coverage` on every input to observe the branch coverage of your | ||
code - but we know that there are many things code can do that aren't captured by | ||
the set of executed branches. `This blog post | ||
<https://blog.foretellix.com/2016/12/23/verification-coverage-and-maximization-the-big-picture/>`__ | ||
gives a good overview of coverage-driven verification workflows. | ||
|
||
We therefore treat each :func:`hypothesis.event` as a "virtual" branch - while it's | ||
not part of the control-flow graph, we keep track of inputs which produced each | ||
observed event in the same way that we track the inputs which produce each branch. | ||
|
||
You can therefore use the :func:`~hypothesis.event` function in your tests to | ||
mark out categories of behaviour, boundary conditions, and so on, and then let the | ||
fuzzer exploit that to generate more diverse and better-targeted inputs. | ||
And as a bonus, you'll get useful summary statistics when running Hypothesis! | ||
|
||
|
||
|
||
The Hypothesis database | ||
----------------------- | ||
|
||
The Hypothesis database forms the basis of HypoFuzz workflows: failing examples | ||
can be reproduced automatically just by running the tests - because those inputs | ||
are added to the database and replayed by Hypothesis itself. | ||
|
||
It is therefore critical that the fuzzer is using the *same* database as | ||
Hypothesis - regardless of how or where you run it. | ||
|
||
:hydocs:`Hypothesis' default database <database.html>` is designed to persist | ||
failing examples on a single developer's machine, and does so via the filesystem. | ||
If you want to share the database between your team members and your CI server | ||
though, this isn't going to work so well - either set the | ||
:obj:`~hypothesis:hypothesis.settings.database` setting to an | ||
:class:`~hypothesis:hypothesis.database.ExampleDatabase` backed by a network | ||
datastore, or use a :class:`~hypothesis:hypothesis.database.DirectoryBasedExampleDatabase` | ||
pointed to a shared filesystem. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,109 @@ | ||
Features | ||
======== | ||
|
||
The core principle of HypoFuzz is that it should be effortless to adopt: | ||
if you have Hypothesis tests, everything else is automatic. If you're | ||
curious about what that "everything else" involves, this page is for you. | ||
|
||
.. contents:: | ||
:local: | ||
|
||
|
||
Collecting tests | ||
---------------- | ||
|
||
HypoFuzz uses :pypi:`pytest` to collect the test functions to fuzz, | ||
with almost identical command-line interfaces. If you're using | ||
:command:`pytest ...` to run your tests, :command:`hypothesis fuzz -- ...` | ||
will fuzz them. | ||
|
||
Note that tests which use `pytest fixtures <https://docs.pytest.org/en/stable/fixture.html>`__, | ||
including ``autouse`` fixtures, are not collected as they may behave | ||
differently outside of the pytest runtime. We recommend using a context | ||
manager and the ``with`` statement instead. | ||
|
||
Support for other test runners, such as :mod:`python:unittest`, | ||
is :doc:`on our roadmap <roadmap>`. | ||
|
||
|
||
Execution model | ||
--------------- | ||
|
||
HypoFuzz runs as one or more worker processes, by default one per available | ||
core, and an additional process serving the live dashboard as a website. | ||
|
||
In each worker process, HypoFuzz prioritizes tests which discover new coverage, | ||
which maximises the rate of discovery and therefore minimises the time taken | ||
to cover each branch in your code. This adaptive approach is one of HypoFuzz's | ||
advantages over other fuzzing workflows - and the reason you can apply it to | ||
a whole test suite at a time. | ||
|
||
|
||
HypoFuzz dashboard | ||
------------------ | ||
|
||
The HypoFuzz dashboard - `online demo here <../../example-dashboard/>`__ - shows | ||
the current state of the fuzzing campaign overall, with a sub-page for each test | ||
to show more information. | ||
|
||
|
||
Fuzzer details | ||
-------------- | ||
|
||
HypoFuzz is, compared to other fuzzers in the literature, a bizzare mixture of | ||
every technique that seems to work. Instead of being based on "one brilliant | ||
idea" (oversimplifying, AFL = "coverage-guided mutation", :cite:`AFLFast` | ||
= "bias towards rare branches", etc.), we have a single simple goal: | ||
*fuzzing your property-based test suite should be effortless*. | ||
|
||
Because HypoFuzz is designed to exploit features that already exist in Hypothesis, | ||
you *can* write tests which are designed to be fuzzed, but idiomatic ``@given`` | ||
tests already work just fine. | ||
|
||
|
||
Basic design | ||
~~~~~~~~~~~~ | ||
|
||
It's a standard feedback-directed greybox fuzzer. The interesting parts are | ||
|
||
1. HypoFuzz tests Python code, not native executables | ||
2. we exploit property-based tests to detect semantic bugs, not just crashes | ||
3. we use Hypothesis to generate highly-structured and typically valid data | ||
4. we leverage a wider variety of feedbacks than most fuzzers | ||
5. we fuzz *very many* more targets than most fuzzing campaigns | ||
|
||
|
||
Corpus distillation | ||
~~~~~~~~~~~~~~~~~~~ | ||
|
||
We exploit Hypothesis' world-class test-case reduction logic ("shrinking") to | ||
maintain a seed pool of minimal covering examples for each branch - or other | ||
reason to retain a seed. | ||
|
||
Those other reasons include user-defined labels via :func:`hypothesis:hypothesis.event`, | ||
real-valued metrics with :func:`hypothesis:hypothesis.target`, | ||
and more to come. | ||
|
||
|
||
Mutation logic | ||
~~~~~~~~~~~~~~ | ||
|
||
The mutation logic is minimum-viable at the moment. It works shockingly well, | ||
thanks to Hypothesis' input structure, but substantial improvements are on the | ||
roadmap. | ||
|
||
|
||
Ensemble fuzzing | ||
~~~~~~~~~~~~~~~~ | ||
|
||
HypoFuzz natively supports ensemble fuzzing :cite:`EnFuzz`, by periodically loading | ||
any new examples from the database. This works in ``--unsafe`` mode, where each | ||
test function might run in multiple fuzzer processes at the same time, and with | ||
other fuzzer tools leveraging e.g. the `.hypothesis.fuzz_one_input | ||
<https://hypothesis.readthedocs.io/en/latest/details.html#use-with-external-fuzzers>`__ | ||
hook. | ||
|
||
Ensemble fuzzing can also be modelled as a mixture of the ensembled behaviours, | ||
and HypoFuzz therefore attempts to run an *adaptive* mixture of all the useful | ||
behaviours we can implement. To the extent that this works, we get the benefits | ||
of ensembling and consume the minimum possible resources to required to do so. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,23 @@ | ||
HypoFuzz documentation | ||
====================== | ||
|
||
:pypi:`Hypothesis` is free and open-source software, and always will be. | ||
It's proven itself invaluable for everyone - from students to high-school | ||
teachers, astrophysicists to data scientists, web developers to systems | ||
programmers, and even for hardware designers. | ||
|
||
HypoFuzz builds on that success: if you have Hypothesis tests, HypoFuzz | ||
gives you a fantastic way to spend CPU time - instead of engineering time - | ||
to find bugs early in your development cycle. | ||
|
||
|
||
.. toctree:: | ||
:maxdepth: 2 | ||
|
||
Main website <https://hypofuzz.com/> | ||
quickstart | ||
features | ||
configuration | ||
roadmap | ||
literature | ||
changelog |
Oops, something went wrong.