DeepXplore: Automated Whitebox Testing of Deep Learning Systems
By Kexin Pei, Yinzhi Cao, Junfeng Yang, Suman Sekhar Jana. 2017
1,541 citations recorded by the source corpus.
Deep learning (DL) systems are increasingly deployed in safety- and security-critical domains including self-driving cars and malware detection, where the correctness and predictability of a system's behavior for corner case inputs are of great importance. Existing DL testing depends heavily on manually labeled data and therefore often fails to expose erroneous behaviors for rare inputs. We design, implement, and evaluate DeepXplore, the first whitebox framework for systematically testing real-world DL systems. First, we introduce neuron coverage for systematically measuring the parts of a DL system exercised by test inputs. Next, we leverage multiple DL systems with similar functionality as cross-referencing oracles to avoid manual checking. Finally, we demonstrate how finding inputs for DL systems that both trigger many differential behaviors and achieve high neuron coverage can be represented as a joint optimization problem and solved efficiently using gradient-based search techniques. DeepXplore efficiently finds thousands of incorrect corner case behaviors (e.g., self-driving cars crashing into guard rails and malware masquerading as benign software) in state-of-the-art DL models with thousands of neurons trained on five popular datasets including ImageNet and Udacity self-driving challenge data. For all tested DL models, on average, DeepXplore generated one test input demonstrating incorrect behavior within one second while running only on a commodity laptop. We further show that the test inputs generated by DeepXplore can also be used to retrain the corresponding DL model to improve the model's accuracy by up to 3%.
Subfields: Artificial Intelligence, Safety, Robustness & Interpretability, Signal Processing, Software Engineering
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Percy Liang
- Mohit Bansal
- Sergey Levine
- Yoshua Bengio
- Bo Li
- Quoc V. Le
- Luke S. Zettlemoyer
- Jianfeng Gao
In Safety, Robustness & Interpretability
- Nicholas Carlini
- Bo Li
- Suman Sekhar Jana
- Florian Tramèr
- Matthias Hein
- Cho-Jui Hsieh
- Huan Zhang
- Nicolas Papernot
In Signal Processing
- Bo Li
- Nicholas Carlini
- Battista Biggio
- Fabio Roli
- Dawn Song
- Suman Sekhar Jana
- Florian Tramèr
- Ian J. Goodfellow
In Software Engineering
- Suman Sekhar Jana
- Yang Liu
- Kexin Pei
- Bo Li
- Ian J. Goodfellow
- Battista Biggio
- Fabio Roli
- Aditya V. Thakur
Citations in the corpus
- Wild Patterns: Ten Years After the Rise of Adversarial Machine Learning (2017, methodology): “Other evaluation methodologies leverage ideas from the field of software testing [81].”
- Efficient Formal Safety Analysis of Neural Networks (2018, methodology): “∗ in [23] 458 21 21 Drebin_FC2 210 <545334, 200, 10, 2> 437 22 41 Drebin_FC3 400 <545334, 200, 200, 2> 297 27 176”
- AI2: Safety and Robustness Certification of Neural Networks with Abstract Interpretation (2018, background): “, [32]) perturbs an image by changing all pixels above the threshold 1− δ to the brightest possible value (in Fig.”
- Formal Security Analysis of Neural Networks using Symbolic Intervals (2018, methodology): “Later, many efficient variants have been proposed to find a diverse category of adversarial examples [7, 33, 38].”
- Explainable Recommendation: A Survey and New Perspectives (2018, background): “Early approaches to personalized recommender systems mostly focused on content-based recommendation or collaborative filtering (CF) based recommendation (Ricci et al. [2011]).”
- Adversarial Examples: Attacks and Defenses for Deep Learning (2017, background): “1) Data Incompletion: One assumption is that adversarial examples are of low probability and low test coverage of corner cases in the testing data set [8], [130].”
- Generating Natural Adversarial Examples (2017, background): “[19] show that minimal changes in the lighting conditions can fool automated-driving systems, a behavior adversarial examples are unable to discover.”
- DeepTest: Automated Testing of Deep-Neural-Network-Driven Autonomous Cars (2017, methodology): “defined neuron coverage only for CNNs [69].”
- Towards Practical Verification of Machine Learning: The Case of Computer Vision Systems (2017, background): “Failing exhaustive input verification, an alternative approach is to take an existing input, alter it, and check or enhance the ML model on the altered input [8], [16], [23], [27], [30], [32], [33], [37], [41], [44], [46], [51].”
- Seeing is Not Believing: Camouflage Attacks on Image Scaling Algorithms (2019, methodology): “Recent research work, such as the sample applications used in DeepXplore [25], also shows that the resizing operation is commonly used in self-driving applications to adjust original video frames’ size to the input size of models.”
- Using Machine Learning Safely in Automotive Software: An Assessment and Adaption of Software Process Requirements in ISO 26262 (2018, background): “Recently, structural coverage metrics have been proposed for DNNs including neuron coverage [66] and adapted version of MC/DC coverage [65].”
- TensorFuzz: Debugging Neural Networks with Coverage-Guided Fuzzing (2018, methodology): “We survey these proposals here: Pei et al. [35] introduce the metric of neuron coverage for a neural network with rectified linear units (ReLus) as the activation functions.”
- DeepMutation: Mutation Testing of Deep Learning Systems (2018, methodology): “DeepXplore [12] proposes a white-box differential testing algorithm to systematically generate adversarial examples that cover all neurons in the network.”
- A Survey of End-to-End Driving: Architectures and Training Methods (2020, background): “Also, the diversity of test driving data is almost never sufficiently high to ensure the safety of driving models in all conditions [123].”
- Are aligned neural networks adversarially aligned? (2023, background): “On the other hand, researchers use adversarial robustness as a way to understand the worst-case behavior of some system [Szegedy et al., 2014, Pei et al., 2017].”
- Challenges in Designing Datasets and Validation for Autonomous Driving (2019, methodology): “There have been attempts to generate tests automatically using deep learning like DeepXplore (Pei et al., 2017) and DeepTest (Tian et al.”
- Global Robustness Evaluation of Deep Neural Networks with Provable Guarantees for L0 Norm (2018, methodology): “We compare our tool L0 - TRE with other state-of-the-art test case generation methods, including DeepConcolic 6 [28] and DeepXplore [20].”