Excessive Invariance Causes Adversarial Vulnerability

By Jörn-Henrik Jacobsen, Jens Behrmann, Richard S. Zemel, Matthias Bethge. 2018

174 citations recorded by the source corpus.

Despite their impressive performance, deep neural networks exhibit striking failures on out-of-distribution inputs. One core idea of adversarial example research is to reveal neural network errors under such distribution shifts. We decompose these errors into two complementary sources: sensitivity and invariance. We show deep networks are not only too sensitive to task-irrelevant changes of their input, as is well-known from epsilon-adversarial examples, but are also too invariant to a wide range of task-relevant changes, thus making vast regions in input space vulnerable to adversarial attacks. We show such excessive invariance occurs across various tasks and architecture types. On MNIST and ImageNet one can manipulate the class-specific content of almost any image without changing the hidden activations. We identify an insufficiency of the standard cross-entropy loss as a reason for these failures. Further, we extend this objective based on an information-theoretic analysis so it encourages the model to consider all task-dependent features in its decision. This provides the first approach tailored explicitly to overcome excessive invariance and resulting vulnerabilities.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Safety, Robustness & Interpretability

Citations in the corpus

Shortcut learning in deep neural networks (2020, background): “tainty is incredibly big. To humans only some of these look like guitars, others like patterns (interpretable or abstract) and many more resemble white noise or even look like airplanes, cats or food [8,45,41]. Figure4on the right, for example, highlights a variety of image pairs that have hardly an”
Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness (2020, background): “…the fields of dimensionality reduction (Blum, 2006; Perrault-Joncas, 2017), generative modeling (Lawrence and Candela, 2006; Dinh et al., 2014; 2016; Jacobsen et al., 2018b), and adversarial robustness (Jacobsen et al., 2018a; Ruan et al., 2018; Sokolic et al., 2017; Tsuzuku et al., 2018; Weng et…”
Fundamental Tradeoffs between Invariance and Sensitivity to Adversarial Perturbations (2020, background): “We study a complementary failure mode to sensitivity adversarial examples, called invariance adversarial examples (Jacobsen et al., 2019).”
Exploiting Excessive Invariance caused by Norm-Bounded Adversarial Robustness (2019, methodology): “The cross-entropy loss as well as architectural elements such as ReLU activation functions have been put forward as possible sources of excessive invariance (Jacobsen et al., 2019; Behrmann et al., 2018).”
Flexibly Fair Representation Learning by Disentanglement (2019, background): “Designing invariant representations with non-variational objectives has also been explored, including in reversible models (Ardizzone et al., 2019; Jacobsen et al., 2019).”
A Fourier Perspective on Model Robustness in Computer Vision (2019, background): “As a simple example of this principle, consider Figure 8 in [19].”
Residual Flows for Invertible Generative Modeling (2019, methodology): “In applications, they have been applied to study adversarial robustness (Jacobsen et al., 2019) and are used to train hybrid models with both generative and classification capabilities (Nalisnick et al.”
Improving Robustness Without Sacrificing Accuracy with Patch Gaussian Augmentation (2019, background): “For instance, networks exhibit excessive invariance to visual features [29], texture bias [15], sensitivity to worst-case (adversarial) perturbations [19], and a propensity to rely solely on non-robust, but highly predictive features for classiﬁcation [11, 28].”
A Disentangling Invertible Interpretation Network for Explaining Latent Representations (2020, methodology): “Invertible neural networks [5, 6, 19, 22] have been used to get a better understanding of adversarial attacks [18].”
Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations (2022, background): “[45] also show synthetic and natural examples, where neural networks ignore relevant features, and Scimeca et al.”
Gradient Starvation: A Learning Proclivity in Neural Networks (2020, background): “Interestingly, [73], [72] and [51] draw a similar conclusion and argue that “an insufficiency of the cross-entropy loss” causes excessive invariances to predictive features.”
Do Wider Neural Networks Really Help Adversarial Robustness? (2020, background): “Recently, [34, 35] pointed out that adversarial examples are generated from the non-robust or invariant features hidden in the training data.”
MintNet: Building Invertible Neural Networks with Masked Convolutions (2019, methodology): “Invertible classifiers are arguably more interpretable, because a prediction can be traced down by inverting latent representations [15, 14].”
Understanding the Limitations of Conditional Generative Models (2019, unknown): “As the contribution of the conditional class information is negligible for the data log likelihood, we choose to model it in its distinct subspace, as proposed by Jacobsen et al. (2019).”
The Pitfalls of Simplicity Bias in Neural Networks (2020, background): “[34] show that NNs exhibit invariance to large label-relevant perturbations.”