Fork me on GitHub

A ground-truth binary fuzzing benchmark suite based on real programs with real bugs.

Magma: A Ground-Truth Fuzzing Benchmark

Magma is a collection of open-source libraries with widespread usage and a long history of security-critical bugs and vulnerabilities. In light of the need for better fuzzer evaluation, we front-ported bugs from previous bug reports to the latest versions of these libraries.

For each ported bug, we added in-line (source-code-level) instrumentation to collect ground-truth information about bugs reached (buggy code executed) and triggered (fault condition satisfied by input). This instrumentation allows a monitoring utility to measure fuzzer progress in real time.

Magma also includes the captain toolset which facilitates the process of building Magma targets and running campaigns.

Check out a sample Magma report and read the paper preprint. Questions, comments, and feedback are welcome!

Overview

Included Libraries

We selected a handful of diverse targets to include in the initial version of Magma. These targets were chosen from the Google OSS-Fuzz list of supported projects which are actively updated and developed:

  1. libpng
  2. libtiff
  3. libxml2
  4. poppler
  5. openssl
  6. sqlite3
  7. php

For each library, we build at least one executable program that consumes an input file and feeds it to the instrumented library. While these programs are not guaranteed to maximize library code coverage, they have proven useful as fuzz targets, since they are used by OSS-Fuzz as libFuzzer/AFL stubs, and a majority of the reports for front-ported bugs in Magma mention these programs in the Proof-of-Concept to reproduce the bugs.