DLFuzz: differential fuzzing testing of deep learning systems

By Jianmin Guo, Yu Jiang, Yue Zhao, Quan Chen, Jiaguang Sun. ESEC/SIGSOFT FSE, 2018

317 citations recorded by the source corpus.

Deep learning (DL) systems are increasingly applied to safety-critical domains such as autonomous driving cars. It is of significant importance to ensure the reliability and robustness of DL systems. Existing testing methodologies always fail to include rare inputs in the testing dataset and exhibit low neuron coverage. In this paper, we propose DLFuzz, the first differential fuzzing testing framework to guide DL systems exposing incorrect behaviors. DLFuzz keeps minutely mutating the input to maximize the neuron coverage and the prediction difference between the original input and the mutated input, without manual labeling effort or cross-referencing oracles from other DL systems with the same functionality. We present empirical evaluations on two well-known datasets to demonstrate its efficiency. Compared with DeepXplore, the state-of-the-art DL whitebox testing framework, DLFuzz does not require extra efforts to find similar functional DL systems for cross-referencing check, but could generate 338.59% more adversarial inputs with 89.82% smaller perturbations, averagely obtain 2.86% higher neuron coverage, and save 20.11% time consumption.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Software

View Paper: https://www.semanticscholar.org/paper/1935ad0dbc42578347b19807d6c8d6d76692d2d7

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xiaowei Huang
2. Wenjie Ruan
3. Daniel Kroening
4. Youcheng Sun
5. Min Wu
6. Yang Liu
7. James Sharp
8. Jie M. Zhang

In Computer Vision and Pattern Recognition

1. Min Wu
2. Daniel Kroening
3. Youcheng Sun
4. Wenjie Ruan
5. Xiaowei Huang
6. Yang Liu
7. James Sharp
8. Mark Harman

In Software

1. Jie M. Zhang
2. Yang Liu
3. Lei Ma
4. Manuel Egele
5. Daniel Kroening
6. Mark Harman
7. Xinping Yi
8. James Sharp

Citations in the corpus

• The Art, Science, and Engineering of Fuzzing: A Survey (2018, background): “ing [140], [232], kernel testing [226], [204], [193], representation dependence testing [125], function detection [237], robustness evaluation [233], exploit development [115], GUI testing [206], signature generation [75], penetration testing [84], [162], embedded devices [161], and neural network t”
• A survey of safety and trustworthiness of deep neural networks: Verification, testing, adversarial attack and defence, and interpretability (2018, methodology): “Moreover, DLFuzz [Guo et al., 2018] is a differential fuzzing testing framework.”
• Machine Learning Testing: Survey, Landscapes and Horizons (2019, methodology): “The preliminary evaluation on MNIST and ImageNet shows that compared with DeepXplore, DLFuzz is able to generate 135% to 584.62% more inputs with 20.11% less time consumption.”