Robustness of classifiers: from adversarial to random noise

By Alhussein Fawzi, Seyed-Mohsen Moosavi-Dezfooli, Pascal Frossard. 2016

405 citations recorded by the source corpus.

Several recent works have shown that state-of-the-art classifiers are vulnerable to worst-case (i.e., adversarial) perturbations of the datapoints. On the other hand, it has been empirically observed that these same classifiers are relatively robust to random noise. In this paper, we propose to study a \textit{semi-random} noise regime that generalizes both the random and worst-case noise regimes. We propose the first quantitative analysis of the robustness of nonlinear classifiers in this general noise regime. We establish precise theoretical bounds on the robustness of classifiers in this general regime, which depend on the curvature of the classifier's decision boundary. Our bounds confirm and quantify the empirical observations that classifiers satisfying curvature constraints are robust to random noise. Moreover, we quantify the robustness of classifiers in terms of the subspace dimension in the semi-random noise regime, and show that our bounds remarkably interpolate between the worst-case and random noise regimes. We perform experiments and show that the derived bounds provide very accurate estimates when applied to various state-of-the-art deep neural networks and datasets. This result suggests bounds on the curvature of the classifiers' decision boundaries that we support experimentally, and more generally offers important insights onto the geometry of high dimensional classification problems.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

Adversarial Examples Are Not Bugs, They Are Features (2019, background): “Fawzi, Moosavi-Dezfooli, and Frossard [FMF16] and Ford et al. [For+19] argue that the adversarial robustness of a classiﬁer can be directly connected to its robustness under (appropriately scaled) random noise.”
Deep Variational Information Bottleneck (2017, methodology): “This setup (which is identical to our experiments) induces a classiﬁer which is bounded by a quadratic function, which is interesting because the theoretical framework Fawzi et al. (2016) proves that quadratic classiﬁers have greater capacity for adversarial robustness than linear functions.”
Delving into Transferable Adversarial Examples and Black-box Attacks (2016, background): “Prior works also try to understand the geometic properties of adversarial examples theoretically (Fawzi et al. (2016)) or empirically (Goodfellow et al. (2014)).”
Certified Adversarial Robustness via Randomized Smoothing (2019, methodology): “The robustness of neural networks to random noise has been analyzed both theoretically (Fawzi et al., 2016; Franceschi et al., 2018) and empirically (Dodge & Karam, 2017).”
Theoretical evidence for adversarial robustness through randomization: the case of the Exponential family (2019, background): “Since the seminal work of [42] exhibiting this intriguing phenomenon in the context of deep learning, a wealth of results have been published on designing attacks [18, 34, 32, 23, 6, 31] and defenses [18, 35, 20, 29, 39, 27]), or on trying to understand the very nature of this phenomenon [17, 40, 15”
Parseval Networks: Improving Robustness to Adversarial Examples (2017, background): “…to the magnitude of the hidden activations gradient with respect to their inputs (Szegedy et al., 2014), they also empirically assessed that standard regularization schemes such as weight decay or training with random noise do not solve the problem (Goodfellow et al., 2015; Fawzi et al., 2016).”
Countering Adversarial Images using Input Transformations (2018, neutral): “The robustness of machine learning models to adversarial examples depends both on the properties of the model ( i.e. , Lipschitzness) and on the nature of the problem considered, e.g. , on the input dimensionality and the Bayes error of the problem (Fawzi et al., 2015; 2016).”
Adversarial Examples for Semantic Segmentation and Object Detection (2017, background): “Given an image and the recognition targets (proposals and/or pixels), DAG generates an adversarial perturbation which is aimed at confusing as many targets as possible.”
Adversarial Examples, Uncertainty, and Transfer Testing Robustness in Gaussian Process Hybrid Deep Networks (2017, background): “What is also concerning is that current vulnerable classifiers are also vulnerable to attacks that only operate on a smaller randomly chosen image subspace [16].”
Detecting Adversarial Samples from Artifacts (2017, background): “Although DNNs are known to be robust to noisy inputs (Fawzi et al., 2016), they have been shown to be vulnerable to specially-crafted adversarial samples (Szegedy et al., 2014; Goodfellow et al., 2015).”
Robustness May Be at Odds with Accuracy (2018, background): “Fawzi et al. (2018b) prove upper bounds on the robustness of classifiers and exhibit a standard vs.”
Adversarially Robust Generalization Requires More Data (2018, background): “[18] relate the robustness of linear and non-linear classifiers to adversarial and (semi-) random perturbations.”
Threat of Adversarial Attacks on Deep Learning in Computer Vision: A Survey (2018, background): “Flatness of decision boundaries [69], large local curvature of the decision boundaries [70] and low flexibility of the networks [71] are some more examples of the viewpoints on the existence of adversarial examples that do not perfectly align with each other.”
On Evaluating Adversarial Robustness (2019, background): “Adversarially robust models tend to be more resistant to random noise compared to their standard counterparts (Fawzi et al., 2016).”
ConvNets and ImageNet Beyond Accuracy: Explanations, Bias Detection, Adversarial Examples and Model Criticism (2017, background): “Several studies have subsequently analyzed the phenomenon [11, 40, 12, 12] and various approaches have been proposed to improve the robustness of neural networks [34, 8, 44, 46, 19].”
Adversarial vulnerability for any classifier (2018, background): “The dependence on K captured in our bounds is in contrast to previous bounds that showed decreasing probability of fooling the classifier, for larger number of classes [20].”
Adversarial Examples Are a Natural Consequence of Test Error in Noise (2019, methodology): “This observation was also made in Fawzi et al. (2016; 2018b).”