Evaluating Fuzz Testing
By George Klees, Andrew Ruef, Benji Cooper, Shiyi Wei, Michael W. Hicks. Conference on Computer and Communications Security, 2018
785 citations recorded by the source corpus.
Fuzz testing has enjoyed great success at discovering security critical bugs in real software. Recently, researchers have devoted significant effort to devising new fuzzing techniques, strategies, and algorithms. Such new ideas are primarily evaluated experimentally so an important question is: What experimental setup is needed to produce trustworthy results? We surveyed the recent research literature and assessed the experimental evaluations carried out by 32 fuzzing papers. We found problems in every evaluation we considered. We then performed our own extensive experimental evaluation using an existing fuzzer. Our results showed that the general problems we found in existing experimental evaluations can indeed translate to actual wrong or misleading assessments. We conclude with some guidelines that we hope will help improve experimental evaluations of fuzz testing algorithms, making reported results more robust.
Subfields: Software, Signal Processing
View Paper: https://www.semanticscholar.org/paper/76655792a9c3ebcbc64d2665ecf5bba81966e478
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Software
- Yinlin Deng
- Chun Xia
- Lingming Zhang
- Haoran Peng
- Chenyuan Yang
- Yang Xiang
- Sheng Wen
- Seyit Ahmet Camtepe
In Signal Processing
- Sang Kil Cha
- Michael W. Hicks
- Sheng Wen
- Chun Xia
- Abhik Roychoudhury
- Seyit Ahmet Camtepe
- Xiaogang Zhu
- Lingming Zhang
Citations in the corpus
- Large Language Model guided Protocol Fuzzing (2024, positive): “…the utility of Large Language Models (LLMs) for tackling the challenges of mutation-based protocol fuzzing of text-based network protocols, we seek to answer the following questions: To answer these questions, we follow the recommended experimental design for fuzzing experiments [26], [10].”
- Large Language Models Are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Models (2022, methodology): “for RQ2, we randomly sample 100 public APIs in each library and conduct the ablation study experiments for ve times and report the average following prior work [37].”
- Fuzzing: A Survey for Roadmap (2022, methodology): “A proper evaluation includes efective experimental corpus [215], fair evaluation environment/platform [30, 104, 126], reasonable fuzzing time [17, 20], and comprehensive comparison metrics [96, 104].”
- Build It, Break It, Fix It: Contesting Secure Development (2016, background): “[39] show that fuzzers are poor at identifying which underlying bugs cause crashing inputs.”
- The Art, Science, and Engineering of Fuzzing: A Survey (2018, background): “[131] recently found that there has been no coherent way of evaluating fuzzing techniques, which can”
- sFuzz: An Efficient Adaptive Fuzzer for Solidity Smart Contracts (2020, background): “Fuzzing is one of themost popularmethods to create test cases [20].”
- Superion: Grammar-Aware Greybox Fuzzing (2018, neutral): “Third, we did not have a statistical sig-nificance analysis argued by Klees et al. [38].”