Measuring Neural Net Robustness with Constraints

By Osbert Bastani, Yani Ioannou, Leonidas Lampropoulos, Dimitrios Vytiniotis, Aditya V. Nori, Antonio Criminisi. 2016

451 citations recorded by the source corpus.

Despite having high accuracy, neural nets have been shown to be susceptible to adversarial examples, where a small perturbation to an input can cause it to become mislabeled. We propose metrics for measuring the robustness of a neural net and devise a novel algorithm for approximating these metrics based on an encoding of robustness as a linear program. We show how our metrics can be used to evaluate the robustness of deep neural nets with experiments on the MNIST and CIFAR-10 datasets. Our algorithm generates more informative estimates of robustness metrics compared to estimates based on existing algorithms. Furthermore, we show how existing approaches to improving robustness "overfit" to adversarial examples generated using a specific algorithm. Finally, we show that our techniques can be used to additionally improve neural net robustness both according to the metrics that we propose, but also according to previously proposed metrics.

Subfields: Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Safety, Robustness & Interpretability

Citations in the corpus

Benchmarking Neural Network Robustness to Common Corruptions and Perturbations (2019, background): “Its popularity has often led “adversarial robustness” to become interchangeable with “robustness” in the literature (Bastani et al., 2016; Rauber et al., 2017).”
Towards Evaluating the Robustness of Neural Networks (2016, background): “…examples not only limits their potential application settings, but entirely defeats its purpose: an adversary who is able to make only slight modifications to a malware file that cause it to remain malware, but become classified as benign, has entirely defeated the malware classifier [8], [14].”
AI2: Safety and Robustness Certification of Neural Networks with Abstract Interpretation (2018, background): “Others have developed approaches that can show non-robustness by underapproximating neural network behaviors [1] or methods that decide robustness of small fully connected feedforward networks [21].”
Evaluating the Robustness of Neural Networks: An Extreme Value Theory Approach (2018, neutral): “For example, Bastani et al. (2016) proposed a linear programming (LP) formulation to ﬁnd adversarial examples and use the distortions as the robustness metric.”
Certified Defenses against Adversarial Examples (2018, background): “Bastani et al. (2016) provide tractable certiﬁcates but require (cid:15) to be small enough to ensure that the entire (cid:96) ∞ ball around an input lies within the same linear region.”
Verifying Neural Networks with Mixed Integer Programming (2017, neutral): “Another approach is to attempt to detect adversarial examples by identifying some properties on which they differ from normal input [1, 9].”
Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methods (2017, background): “There has been a significant amount of work studying methods to construct adversarial examples [5, 8, 13, 26, 30, 39] and to make networks robust against adversarial examples [3, 16, 20, 21, 31, 33, 34, 41].”
DeepXplore: Automated Whitebox Testing of Deep Learning Systems (2017, background): “This observation has inspired several researchers to try to improve the robustness and reliability of DNNs [9, 13, 17, 30, 32, 46, 51, 53, 62, 84, 89, 90].”
Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networks (2017, methodology): “As observed by us and others [3, 10], this theoretical worst-case behavior is also seen in practice, and hence this approach is practical only for very small networks.”
Adversarial Examples: Attacks and Defenses for Deep Learning (2017, background): “[15], [23], [62], [65] conducted initial researches on the evaluation.”
Universal Adversarial Perturbations (2016, neutral): “Such carefully crafted perturbations are either estimated by solving an optimization problem [19, 11, 1] or through one step of gradient ascent [5], and result in a perturbation that fools a speciﬁc data point.”
Verifiable Reinforcement Learning via Policy Extraction (2018, background): “…decision tree policy that plays Atari Pong (on a symbolic abstraction of the state space rather than from pixels 2 ) [22] and verify its robustness [6, 19], (ii) we learn a decision tree policy to play a toy game based on Pong, and prove that it never loses (the difﬁculty doing so for Atari Pong…”
Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networks (2018, background): “[4] encoded networks to linear programs, Katz et al.”
Formal Guarantees on the Robustness of a Classifier against Adversarial Manipulation (2017, background): “While countermeasures have been proposed [8, 7, 26, 18, 12, 2], none of them provides a guarantee”
The Space of Transferable Adversarial Examples (2017, methodology): “This hypothesis is pessimistic: it implies that a model cannot be secure against adversarial examples transferred from other models (i.e, black-box attacks) unless it is also robust to adversarial examples crafted with knowledge of the model’s parameters (i.e., white-box attacks).”
Towards Proving the Adversarial Robustness of Deep Neural Networks (2017, background): “We begin with a common definition for local adversarial robustness [2, 9, 13]:”