Feature-Guided Black-Box Safety Testing of Deep Neural Networks

By Matthew Wicker, Xiaowei Huang, Marta Z. Kwiatkowska. 2017

247 citations recorded by the source corpus.

Despite the improved accuracy of deep neural networks, the discovery of adversarial examples has raised serious safety concerns. Most existing approaches for crafting adversarial examples necessitate some knowledge (architecture, parameters, etc.) of the network at hand. In this paper, we focus on image classifiers and propose a feature-guided black-box approach to test the safety of deep neural networks that requires no such knowledge. Our algorithm employs object detection techniques such as SIFT (Scale Invariant Feature Transform) to extract features from an image. These features are converted into a mutable saliency distribution, where high probability is assigned to pixels that affect the composition of the image with respect to the human visual system. We formulate the crafting of adversarial examples as a two-player turn-based stochastic game, where the first player's objective is to minimise the distance to an adversarial example by manipulating the features, and the second player can be cooperative, adversarial, or random. We show that, theoretically, the two-player game can con- verge to the optimal strategy, and that the optimal strategy represents a globally minimal adversarial image. For Lipschitz networks, we also identify conditions that provide safety guarantees that no adversarial examples exist. Using Monte Carlo tree search we gradually explore the game state space to search for adversarial examples. Our experiments show that, despite the black-box setting, manipulations guided by a perception-based saliency distribution are competitive with state-of-the-art methods that rely on white-box saliency matrices or sophisticated optimization procedures. Finally, we show how our method can be used to evaluate robustness of neural networks in safety-critical applications such as traffic sign recognition in self-driving cars.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Reachability Analysis of Deep Neural Networks with Provable Guarantees (2018, methodology): “ reducing the problem to a constraint satisfaction problem that can be solved by MILP [9, 11, 12, 13], SAT [14] or SMT [7, 12] techniques, or applying search algorithms over discretised vector spaces [6, 15]. Even though they are able to achieve guarantees, they suffer from two major weaknesses. Fir”
Abduction-Based Explanations for Machine Learning Models (2018, result): “…efforts to validate the results of ML models (Le-ofante et al. 2018; Ruan, Huang, and Kwiatkowska 2018; Narodytska 2018; Narodytska et al. 2018; Wicker, Huang, and Kwiatkowska 2018; Huang et al. 2017; Katz et al. 2017), but also efforts to explain predictions made by such models (Evans and…”
A Safety Framework for Critical Systems Utilising Deep Neural Networks (2020, methodology): “Formal verification [24,27,66,21,37,64,48,65,49,30] and coverage-guided testing [61,44,59,38,60,62] currently form the two major classes of V&V techniques for DL, from which a collection of evidence may be obtained that supports the partial prior knowledge.”
Bayes-Probe: Distribution-Guided Sampling for Prediction Level Sets (2020, background): “Several other neural network testing paradigms have been proposed (Wicker et al., 2018; Pei et al., 2017; Ma et al., 2018).”
DLFuzz: differential fuzzing testing of deep learning systems (2018, methodology): “Researchers have devoted many efforts to applying software testing to DL testing in both whitebox and blackbox manner [3, 6, 19].”
DeepGauge: Multi-Granularity Testing Criteria for Deep Learning Systems (2018, methodology): “DLV [53] can verify local robustness of DL systems w.”
Machine Learning Testing: Survey, Landscapes and Horizons (2019, methodology): “[93] used feature-guided Monte Carlo Tree Search to identify elements of an image that are most vulnerable to a self-driving system; adversarial examples.”
Concolic Testing for Deep Neural Networks (2018, methodology): “So far, there have been only few attempts to test DNNs systematically [15, 18, 23, 25, 30].”
TensorFuzz: Debugging Neural Networks with Coverage-Guided Fuzzing (2018, methodology): “Wicker et al. [45] perform black box testing of image classiﬁers using image-speciﬁc operations.”
DeepHunter: a coverage-guided fuzz testing framework for deep neural networks (2019, background): “The reliability of DNNs has been investigated by recent work with formal guarantees [18, 25, 37, 39, 45, 47].”
Global Robustness Evaluation of Deep Neural Networks with Provable Guarantees for L0 Norm (2018, methodology): “To mitigate this, a Monte-Carlo tree search has been employed [31].”
Model Agnostic Defence Against Backdoor Attacks in Machine Learning (2019, background): “encies. DeepTest [21] leverages metamorphic relations to discover error cases in DNNs. A recent work DeepGauge [13] measures the test quality and formalises a set of testing criteria for DNNs. SafeCV [26] takes a feature guided black-box approach to verify the safety of DNNs. DeepConcolic [18] perfo”