Skip to content

Commit

Permalink
build docs
Browse files Browse the repository at this point in the history
  • Loading branch information
actions-user committed Feb 19, 2024
1 parent 33a9b10 commit 5af7ba0
Show file tree
Hide file tree
Showing 60 changed files with 8,364 additions and 0 deletions.
4 changes: 4 additions & 0 deletions docs/docs/.buildinfo
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
# Sphinx build info version 1
# This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done.
config: c1fffd718f2a7b6b1b87397abc933d3b
tags: 645f666f9bcd5a90fca523b33c5a78b7
Binary file added docs/docs/.doctrees/changelog.doctree
Binary file not shown.
Binary file added docs/docs/.doctrees/configuration.doctree
Binary file not shown.
Binary file added docs/docs/.doctrees/environment.pickle
Binary file not shown.
Binary file added docs/docs/.doctrees/features.doctree
Binary file not shown.
Binary file added docs/docs/.doctrees/index.doctree
Binary file not shown.
Binary file added docs/docs/.doctrees/literature.doctree
Binary file not shown.
Binary file added docs/docs/.doctrees/quickstart.doctree
Binary file not shown.
Binary file added docs/docs/.doctrees/roadmap.doctree
Binary file not shown.
74 changes: 74 additions & 0 deletions docs/docs/_sources/changelog.md.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
# Changelog
HypoFuzz uses [calendar-based versioning](https://calver.org/), with a
`YY-MM-patch` format.

## 24.02.3
Fixed compatibility with Pytest 8.x ([#32](https://github.com/Zac-HD/hypofuzz/issues/32)).

## 24.02.2
Fixed a dashboard bug ([#31](https://github.com/Zac-HD/hypofuzz/issues/31)).

## 24.02.1
Now requires [Hypothesis 6.93.2](https://hypothesis.readthedocs.io/en/latest/changes.html#v6-93-2)
or later, fixing compatibility with some unstable internals that HypoFuzz hooks into (yes, again).
Also deduplicates the displayed covering examples in the dashboard, when their reprs are identical.

## 23.12.1
Now requires [Hypothesis 6.91](https://hypothesis.readthedocs.io/en/latest/changes.html#v6-91-0)
or later, fixing compatibility with some unstable internals that HypoFuzz hooks into.

## 23.07.1
Various small patches for issues found by fuzzing Hypothesis itself.
Notably, we now try to fuzz functions which require autouse fixtures
even though we don't provide those fixtures - it often works!

## 23.06.1
Hypofuzz now writes ``.patch`` files with failing examples, *and* optionally
with covering examples - including removal of redundant covering examples.

## 23.05.3
Yet another compatibility fix for explain mode, with even _more_ tests for future regressions.
Third time lucky, we hope!

## 23.05.2
Additional compatibility fix, and improved tests to avoid future regressions.

## 23.05.1
Now requires [Hypothesis 6.75.2](https://hypothesis.readthedocs.io/en/latest/changes.html#v6-75-2)
or later, fixing compatibility with some unstable internals that HypoFuzz hooks into.

## 23.04.1
Fixed a bug affecting traceback reporting on Python 3.10+ (#16).

## 22.12.1
Fixed a `NotImplementedError` when HypoFuzz had found a failing input for *every*
test running in some process. Now, we cleanly `sys.exit(1)` instead.

## 22.10.1
First open-source release! Also improves database handling and adds
a timelimit to shrinking to better handle pathological cases.

## 22.07.1
Fixes compatibility with [Hypothesis 6.49.1](https://hypothesis.readthedocs.io/en/latest/changes.html#v6-49-1)
and later (released July 5th).

## 21.12.2
Web-based time-travel debugging with [PyTrace](https://pytrace.com/) - just
`pip install hypofuzz[pytrace]` and click the link on any failing test!

## 21.12.1
Improved dashboard (again!), better generation/mutation heuristics,
basic residual-risk estimation, free-for-noncommercial-use on PyPI.

## 21.05.1
Improved dashboard, fixed memory leaks, support arbitary `hypothesis.event()`
calls as coverage, performance improvements.

## 20.09.1
Multiprocess fuzzing with seed sharing and corpus distillation,
and basic prioritisation of seeds covering rare branches.

Still very much in alpha, but the fundamentals all work now.

## 20.08.1
First packaged version of HypoFuzz
61 changes: 61 additions & 0 deletions docs/docs/_sources/configuration.rst.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
Configuration
=============

.. contents::
:local:


It's all automatic
------------------

Wherever possible, HypoFuzz is designed to do the right thing without configuration.

The :command:`hypothesis fuzz` CLI accepts arguments to determine parameters like the
number of processes to use, and the port on which to serve the dashboard -
though every one of them is optional - and you can select which tests to fuzz
in the same way you would select them via :pypi:`pytest`.

Once that's all set up though, the fuzzer is not configurable: manual prioritization
of a fuzzing process is considerably more error-prone than allowing the adaptive
scheduler to do its thing, and computer time is much cheaper than yours.



Custom coverage events
----------------------

HypoFuzz runs :pypi:`coverage` on every input to observe the branch coverage of your
code - but we know that there are many things code can do that aren't captured by
the set of executed branches. `This blog post
<https://blog.foretellix.com/2016/12/23/verification-coverage-and-maximization-the-big-picture/>`__
gives a good overview of coverage-driven verification workflows.

We therefore treat each :func:`hypothesis.event` as a "virtual" branch - while it's
not part of the control-flow graph, we keep track of inputs which produced each
observed event in the same way that we track the inputs which produce each branch.

You can therefore use the :func:`~hypothesis.event` function in your tests to
mark out categories of behaviour, boundary conditions, and so on, and then let the
fuzzer exploit that to generate more diverse and better-targeted inputs.
And as a bonus, you'll get useful summary statistics when running Hypothesis!



The Hypothesis database
-----------------------

The Hypothesis database forms the basis of HypoFuzz workflows: failing examples
can be reproduced automatically just by running the tests - because those inputs
are added to the database and replayed by Hypothesis itself.

It is therefore critical that the fuzzer is using the *same* database as
Hypothesis - regardless of how or where you run it.

:hydocs:`Hypothesis' default database <database.html>` is designed to persist
failing examples on a single developer's machine, and does so via the filesystem.
If you want to share the database between your team members and your CI server
though, this isn't going to work so well - either set the
:obj:`~hypothesis:hypothesis.settings.database` setting to an
:class:`~hypothesis:hypothesis.database.ExampleDatabase` backed by a network
datastore, or use a :class:`~hypothesis:hypothesis.database.DirectoryBasedExampleDatabase`
pointed to a shared filesystem.
109 changes: 109 additions & 0 deletions docs/docs/_sources/features.rst.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@
Features
========

The core principle of HypoFuzz is that it should be effortless to adopt:
if you have Hypothesis tests, everything else is automatic. If you're
curious about what that "everything else" involves, this page is for you.

.. contents::
:local:


Collecting tests
----------------

HypoFuzz uses :pypi:`pytest` to collect the test functions to fuzz,
with almost identical command-line interfaces. If you're using
:command:`pytest ...` to run your tests, :command:`hypothesis fuzz -- ...`
will fuzz them.

Note that tests which use `pytest fixtures <https://docs.pytest.org/en/stable/fixture.html>`__,
including ``autouse`` fixtures, are not collected as they may behave
differently outside of the pytest runtime. We recommend using a context
manager and the ``with`` statement instead.

Support for other test runners, such as :mod:`python:unittest`,
is :doc:`on our roadmap <roadmap>`.


Execution model
---------------

HypoFuzz runs as one or more worker processes, by default one per available
core, and an additional process serving the live dashboard as a website.

In each worker process, HypoFuzz prioritizes tests which discover new coverage,
which maximises the rate of discovery and therefore minimises the time taken
to cover each branch in your code. This adaptive approach is one of HypoFuzz's
advantages over other fuzzing workflows - and the reason you can apply it to
a whole test suite at a time.


HypoFuzz dashboard
------------------

The HypoFuzz dashboard - `online demo here <../../example-dashboard/>`__ - shows
the current state of the fuzzing campaign overall, with a sub-page for each test
to show more information.


Fuzzer details
--------------

HypoFuzz is, compared to other fuzzers in the literature, a bizzare mixture of
every technique that seems to work. Instead of being based on "one brilliant
idea" (oversimplifying, AFL = "coverage-guided mutation", :cite:`AFLFast`
= "bias towards rare branches", etc.), we have a single simple goal:
*fuzzing your property-based test suite should be effortless*.

Because HypoFuzz is designed to exploit features that already exist in Hypothesis,
you *can* write tests which are designed to be fuzzed, but idiomatic ``@given``
tests already work just fine.


Basic design
~~~~~~~~~~~~

It's a standard feedback-directed greybox fuzzer. The interesting parts are

1. HypoFuzz tests Python code, not native executables
2. we exploit property-based tests to detect semantic bugs, not just crashes
3. we use Hypothesis to generate highly-structured and typically valid data
4. we leverage a wider variety of feedbacks than most fuzzers
5. we fuzz *very many* more targets than most fuzzing campaigns


Corpus distillation
~~~~~~~~~~~~~~~~~~~

We exploit Hypothesis' world-class test-case reduction logic ("shrinking") to
maintain a seed pool of minimal covering examples for each branch - or other
reason to retain a seed.

Those other reasons include user-defined labels via :func:`hypothesis:hypothesis.event`,
real-valued metrics with :func:`hypothesis:hypothesis.target`,
and more to come.


Mutation logic
~~~~~~~~~~~~~~

The mutation logic is minimum-viable at the moment. It works shockingly well,
thanks to Hypothesis' input structure, but substantial improvements are on the
roadmap.


Ensemble fuzzing
~~~~~~~~~~~~~~~~

HypoFuzz natively supports ensemble fuzzing :cite:`EnFuzz`, by periodically loading
any new examples from the database. This works in ``--unsafe`` mode, where each
test function might run in multiple fuzzer processes at the same time, and with
other fuzzer tools leveraging e.g. the `.hypothesis.fuzz_one_input
<https://hypothesis.readthedocs.io/en/latest/details.html#use-with-external-fuzzers>`__
hook.

Ensemble fuzzing can also be modelled as a mixture of the ensembled behaviours,
and HypoFuzz therefore attempts to run an *adaptive* mixture of all the useful
behaviours we can implement. To the extent that this works, we get the benefits
of ensembling and consume the minimum possible resources to required to do so.
23 changes: 23 additions & 0 deletions docs/docs/_sources/index.rst.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
HypoFuzz documentation
======================

:pypi:`Hypothesis` is free and open-source software, and always will be.
It's proven itself invaluable for everyone - from students to high-school
teachers, astrophysicists to data scientists, web developers to systems
programmers, and even for hardware designers.

HypoFuzz builds on that success: if you have Hypothesis tests, HypoFuzz
gives you a fantastic way to spend CPU time - instead of engineering time -
to find bugs early in your development cycle.


.. toctree::
:maxdepth: 2

Main website <https://hypofuzz.com/>
quickstart
features
configuration
roadmap
literature
changelog
Loading

0 comments on commit 5af7ba0

Please sign in to comment.