Evaluating Fuzz Testing
https://dl.acm.org/doi/10.1145/3243734.3243804
Method
"We surveyed the recent research literature and assessed the experimental evaluations carried out by 32 fuzzing papers. We found problems in every evaluation we considered. We then performed our own extensive experimental evaluation using an existing fuzzer... after carrying out more than 50000 CPU hours of experiments"
Population
"We examined 32 recently published papers on fuzz testing"
What it does not show
Not evidence that fuzzing fails to find real bugs; both fuzzers found ground-truth-verified defects. It is evidence that the literature’s method for comparing fuzzer algorithms is unreliable. Does not evaluate production continuous-fuzzing services.
George Klees, Andrew Ruef, Benji Cooper, Shiyi Wei, Michael Hicks
Every one of 32 surveyed evaluations had methodological problems. In the reproduction, a statistically significant increase in unique crashing inputs (p=10^-10) corresponded to no significant increase in real distinct bugs (p=0.066) — the standard deduplication proxy overstates the difference between fuzzers.
Tier I: A systematic survey of 32 papers combined with the authors’ own controlled re-evaluation across varied seeds, timeouts and targets, with statistical significance testing and ground-truth bug identification.