Adversarial Examples Are Not Bugs, They Are Features

By Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, Aleksander Madry. 2019

2,151 citations recorded by the source corpus.

Adversarial examples have attracted significant attention in machine learning, but the reasons for their existence and pervasiveness remain unclear. We demonstrate that adversarial examples can be directly attributed to the presence of non-robust features: features derived from patterns in the data distribution that are highly predictive, yet brittle and incomprehensible to humans. After capturing these features within a theoretical framework, we establish their widespread existence in standard datasets. Finally, we present a simple setting where we can rigorously tie the phenomena we observe in practice to a misalignment between the (human-specified) notion of robustness and the inherent geometry of the data.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Safety, Robustness & Interpretability, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Safety, Robustness & Interpretability

In Signal Processing

Citations in the corpus

Learning Transferable Visual Models From Natural Language Supervision (2021, background): “What explains this discrepancy? Various ideas have been suggested and studied (Ilyas et al., 2019; Geirhos et al., 2020).”
Shortcut learning in deep neural networks (2020, background): “In addition to shortcut opportunities that are fairly easy to recognise, deep learning has led to the discovery of much more subtle shortcut features, including high-frequency patterns that are almost invisible to the human eye [32, 33].”
A Neural Scaling Law from the Dimension of the Data Manifold (2020, methodology): “We include data from fully-connected teacher/student experiments, simple CNNs, and GPT-type [RNSS18, RWC + 19] language models (represented as a lower-bound due to large uncertainties with large IDs).”
Adversarial Training for High-Stakes Reliability (2022, methodology): “In addition, contractors took on average 44 minutes (95% CI: [31, 67]) to generate an adversarial example for +tool-assisted without tools, compared to an average of 26 minutes (95% CI: [24, 29]) with our rewrite tool.”
Feature Purification: How Adversarial Training Performs Robust Deep Learning (2020, background): “Our theoretical work is fundamentally different from the aspect of poor statistical concentration over finite-sample data set, and our theorems strongly supports [27] that a well-trained, well-generalizing neural network can still be non-robust to adversarial attacks.”
Fundamental Tradeoffs between Invariance and Sensitivity to Adversarial Perturbations (2020, background): “Sanitization thus enhances “spurious correlations” (Jo & Bengio, 2017; Ilyas et al., 2019) between nonpredictive features and class labels.5 We further assume that the data labeling process introduces some small label noise.6 Specifically, the labeled data distribution D on which we train and…”
Zero-shot Knowledge Transfer via Adversarial Belief Matching (2019, background): “Here we refer the reader to recent work by Ilyas et al. (2019), who isolate adversarial features and show that they are enough to learn classiﬁers that generalize well to real data.”
Explainable machine learning in deployment (2019, background): “The claim of one of these works is that the closest adversarial example should perturb ‘fragile’ features, enabling the model to fit to robust features (indicative of a particular class) [30].”
A Fourier Perspective on Model Robustness in Computer Vision (2019, background): “Simple statistics such as colors, local textures, shapes, even unintuitive high frequency patterns (Ilyas et al., 2019) can all be leveraged in a way to achieve remarkable i.i.d generalization.”
A Wholistic View of Continual Learning with Deep Neural Networks: Forgotten Lessons and the Bridge to Active and Open World Learning (2020, background): “”Adversarial examples are not bugs they are features” [160] takes this one step further and empirically showcases how classes can be distinguishable solely based on noise patterns.”
Identifying Statistical Bias in Dataset Replication (2020, background): “Previous work has characterized cross-dataset generalization(Torralba & Efros, 2011), and explored the impact of synthetic perturbations on generalization, such as adversarial examples (Kurakin et al., 2016; Tsipras et al., 2019; Ilyas et al., 2019; Su et al., 2018) or various other corruption robus”
Improving Robustness Without Sacrificing Accuracy with Patch Gaussian Augmentation (2019, background): “For instance, networks exhibit excessive invariance to visual features [29], texture bias [15], sensitivity to worst-case (adversarial) perturbations [19], and a propensity to rely solely on non-robust, but highly predictive features for classiﬁcation [11, 28].”
Learning Explainable Models Using Attribution Priors (2019, methodology): “Ilyas et al. [13] discuss how classiﬁers exploit non-intuitive features, and conclude that it is unreasonable to expect classiﬁers to behave in a way that is meaningful to humans without explicitly encoding such behavior into the model.”
A Unifying Review of Deep and Shallow Anomaly Detection (2020, background): “Research on the unresolved and fundamental issue of adversarial examples and attacks [502]–[511] is related to AD as well.”
LLM Lies: Hallucinations are not Bugs, but Features as Adversarial Examples (2023, background): “Moreover, [7] has explained that adversarial examples are fundamental feature of deep neural networks.”