Experiment Configuration

• Duration: 1w
• Trials: 10
• Version: v1.0

Fuzz Targets

This report summarizes the results of fuzzing the following targets:

• libpng
• libtiff
• libxml2
• openssl
• php
• poppler
• sqlite3

Evaluated Fuzzers

The fuzzers used in this evaluation are listed below:

• afl
• aflfast
• aflplusplus
• honggfuzz
• moptafl
• symcc_afl

Experiment Summary

Total Unique Bugs Triggered

• Mean and Standard Deviation

  This plot shows the mean number of unique bugs triggered by every fuzzer against every target, and the standard deviation bar, across all campaigns.

• Stastistical Significance

  This matrix summarizes the p-values of the pairwise Mann-Whitney U tests calculated against the total bug count sample sets collected for every fuzzer across all campaigns. Cells with a green shade indicate that the number of bugs triggered by a fuzzer is statistically different.

Mean Survival Time

• Ranking of Bugs and Fuzzers

  This table shows the restricted mean survival time for every bug being reached or triggered over the duration of the campaign, using the Kaplan-Meier non-parametric survival function estimator. Green-shaded cells indicate the best performance (lowest time) for a bug metric across all fuzzers. Yellow-shaded cells indicate the worst performance (highest time) for a bug metric across all fuzzers. Red-shaded cells indicate that the bug survived being reached or triggered by the fuzzer throughout the campaign duration.

color_lens Change Colormap

Fuzzer	moptafl		honggfuzz		aflplusplus		afl		aflfast		symcc_afl
Metric	R	T	R	T	R	T	R	T	R	T	R	T
Bug ID
AAH037	10s	20s	15s	15s	10s	46s	10s	20s	10s	20s	10s	26s
AAH041	15s	21s	15s	15s	15s	48s	15s	20s	15s	20s	15s	30s
AAH003	10s	16s	15s	17s	10s	15s	10s	15s	10s	15s	10s	1m
JCH207	10s	1m	10s	2m	10s	1m	10s	2m	10s	3m	10s	1m
AAH056	15s	14m	15s	19m	15s	19m	15s	16m	15s	14m	15s	17m
AAH015	32s	1m	15s	22s	27s	17m	1m	59m	1m	8m	27m	1h
AAH052	15s	3m	18s	15m	15s	45m	15s	3h	15s	6h	15s	5m
AAH055	15s	40m	15s	4m	20s	3h	4m	4h	20s	1h	4m	2h
AAH020	10s	2h	10s	2h	10s	31m	10s	2h	10s	49m	5s	11h
AAH022	1m	54m	15s	19m	27s	3h	15m	19h	1m	3h	15m	15h
AAH032	15s	3m	10s	2m	15s	1h	15s	2d	15s	15m	15s	1d
AAH017	13h	13h	2d	2d	13h	13h	13h	14h	6h	6h	9h	9h
JCH215	2m	3h	24s	2h	22m	13h	1h	1d	21m	15h	1h	3d
MAE016	10s	1m	10s	15s	10s	5m	10s	2m	10s	6m	1w	1w
MAE014	15s	1h	15s	55m	15s	14m	15s	4m	15s	10m	1w	1w
MAE008	15s	1h	15s	14h	15s	1m	15s	3m	15s	2m	1w	1w
JCH201	15s	17h	15s	5d	15s	20h	15s	11h	15s	13h	15s	14h
JCH232	4h	4h	44m	2h	1d	2d	3d	4d	1d	2d	4d	4d
AAH014	14h	14h	5d	5d	13h	13h	18h	18h	5d	5d	1d	1d
AAH008	15s	1d	15s	3h	15s	5d	15s	1d	15s	5d	15s	2d
JCH209	14m	14m	1w	1w	2d	2d	2d	2d	1d	1d	2d	2d
MAE115	15s	2d	15s	4d	15s	5d	15s	1d	15s	1d	15s	10h
AAH010	1d	1d	22h	1d	14m	5d	14m	1w	5d	6d	19h	2d
JCH228	16h	1d	6h	2d	3d	4d	5d	6d	2d	4d	5d	6d
AAH007	15s	1w	10s	1m	15s	6d	15s	6d	15s	1w	15s	23m
AAH045	20s	3h	20s	21m	20s	1w	20s	1w	20s	6d	20s	6d
MAE104	15s	2d	15s	1w	15s	4d	15s	4d	15s	2d	1w	1w
AAH013	1w	1w	2h	2h	13h	13h	1w	1w	1w	1w	1w	1w
AAH001	15s	6d	15s	12h	15s	1w	15s	6d	15s	1w	15s	3d
AAH024	15s	9h	20s	2d	15s	1w	15s	1w	15s	1w	15s	1w
AAH016	2d	2d	5d	5d	5d	5d	3d	6d	4d	5d	6d	6d
AAH026	15s	3d	15s	15h	15s	1w	15s	1w	15s	1w	15s	1w
JCH226	2d	3d	3h	19h	1w	1w	1w	1w	1w	1w	1w	1w
AAH049	20s	1d	15s	1w	15s	1w	15s	1w	15s	1w	15s	1w
JCH210	30s	2d	34s	6d	1m	1w	1m	1w	1m	1w	1m	1w
AAH035	15s	3d	15s	1w	25s	1w	15s	1w	15s	1w	20s	1w
JCH212	15s	1w	20s	3d	15s	1w	15s	1w	15s	1w	15s	1w
JCH227	2d	5d	4d	6d	1w	1w	1w	1w	1w	1w	1w	1w
AAH025	5d	5d	6d	6d	1w	1w	1w	1w	1w	1w	1w	1w
AAH043	25s	5d	1w	1w	16h	1w	1w	1w	6d	1w	2d	1w
AAH048	20s	1w	15s	5d	35s	1w	20s	1w	20s	1w	20s	1w
AAH050	25s	5d	6d	6d	29s	1w	1d	1w	28s	1w	29s	1w
JCH216	2d	5d	1w	1w	1w	1w	1w	1w	1w	1w	1w	1w
AAH046	3d	6d	1w	1w	1w	1w	1w	1w	1w	1w	1w	1w
AAH042	40s	6d	24s	1w	1m	1w	45s	1w	40s	1w	45s	1w
AAH009	21h	6d	4d	1w	5d	1w	6h	1w	2d	1w	2d	1w
JCH223	16m	6d	32s	1w	1h	1w	5h	1w	2h	1w	11h	1w
AAH029	15s	6d	20s	1w	15s	1w	15s	1w	15s	1w	15s	1w
JCH229	4m	6d	24s	1w	24m	1w	2h	1w	49m	1w	2h	1w
JCH214	35s	1w	54s	1w	1m	1w	40s	1w	35s	1w	42s	1w
MAE111	18s	1w	15s	1w	15s	1w	20s	1w	20s	1w	20s	1w
MAE105	15s	1w	15s	1w	15s	1w	15s	1w	15s	1w	15s	1w
AAH034	15s	1w	15s	1w	15s	1w	15s	1w	15s	1w	15s	1w
MAE103	33s	1w	1m	1w	33s	1w	40s	1w	34s	1w	31s	1w
AAH004	16s	1w	15s	1w	15s	1w	15s	1w	15s	1w	15s	1w
AAH005	15s	1w	15s	1w	15s	1w	15s	1w	15s	1w	15s	1w
AAH051	25s	1w	15s	1w	42s	1w	25s	1w	25s	1w	30s	1w
MAE006	15s	1w	15s	1w	15s	1w	15s	1w	15s	1w	1w	1w
MAE004	15s	1w	15s	1w	15s	1w	15s	1w	15s	1w	1w	1w
JCH233	8m	1w	23m	1w	6h	1w	11h	1w	1h	1w	5h	1w
AAH018	41m	1w	5m	1w	5h	1w	1h	1w	1h	1w	1h	1w
JCH231	21m	1w	35s	1w	2h	1w	5h	1w	5h	1w	11h	1w
JCH230	4m	1w	41s	1w	1h	1w	5h	1w	1h	1w	5h	1w
AAH059	20s	1w	15s	1w	17s	1w	20s	1w	20s	1w	20s	1w
JCH202	15s	1w	15s	1w	15s	1w	15s	1w	15s	1w	15s	1w
MAE114	15s	1w	15s	1w	15s	1w	15s	1w	15s	1w	15s	1w
JCH204	26s	1w	24s	1w	48s	1w	30s	1w	26s	1w	30s	1w
JCH222	1h	1w	39m	1w	4d	1w	6d	1w	2d	1w	5d	1w
JCH220	4m	1w	20s	1w	22m	1w	1h	1w	46m	1w	2h	1w
JCH219	3d	1w	1w	1w	6d	1w	1w	1w	1w	1w	1w	1w
AAH011	15s	1w	15s	1w	15s	1w	15s	1w	15s	1w	15s	1w
AAH047	2d	1w	1w	1w	16h	1w	1w	1w	6d	1w	2d	1w
AAH031	20s	1w	22s	1w	42s	1w	20s	1w	20s	1w	25s	1w
AAH054	10s	1w	10s	1w	10s	1w	10s	1w	10s	1w	10s	1w