Exploiting Excessive Invariance caused by Norm-Bounded Adversarial Robustness

By Jörn-Henrik Jacobsen, Jens Behrmann, Nicholas Carlini, Florian Tramèr, Nicolas Papernot. 2019

48 citations recorded by the source corpus.

Adversarial examples are malicious inputs crafted to cause a model to misclassify them. Their most common instantiation, "perturbation-based" adversarial examples introduce changes to the input that leave its true label unchanged, yet result in a different model prediction. Conversely, "invariance-based" adversarial examples insert changes to the input that leave the model's prediction unaffected despite the underlying input's label having changed. In this paper, we demonstrate that robustness to perturbation-based adversarial examples is not only insufficient for general robustness, but worse, it can also increase vulnerability of the model to invariance-based adversarial examples. In addition to analytical constructions, we empirically study vision classifiers with state-of-the-art robustness to perturbation-based adversaries constrained by an $\ell_p$ norm. We mount attacks that exploit excessive model invariance in directions relevant to the task, which are able to find adversarial examples within the $\ell_p$ ball. In fact, we find that classifiers trained to be $\ell_p$-norm robust are more vulnerable to invariance-based adversarial examples than their undefended counterparts. Excessive invariance is not limited to models trained to be robust to perturbation-based $\ell_p$-norm adversaries. In fact, we argue that the term adversarial example is used to capture a series of model limitations, some of which may not have been discovered yet. Accordingly, we call for a set of precise definitions that taxonomize and address each of these shortcomings in learning.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness (2020, background): “|X prevents the hidden representation from being unnecessarily invariant to the semantically meaningful changes in the input manifold (Jacobsen et al., 2019; van Amersfoort et al., 2020).”
Privacy Risks of Securing Machine Learning Models against Adversarial Examples (2019, background): “Jacobsen et al. [23] further show that robust training with a l p perturbation constraint makes the model more vulnerable to another type of adversarial examples: invariance based attacks that change the semantics of the image but leave the model predictions unchanged.”
Adversarial Training and Robustness for Multiple Perturbations (2019, background): “Despite its idiosyncrasies, MNIST is an interesting case-study as distinct models achieve strong (arguably too strong in some cases [16]) robustness to different lp-perturbations and spatial attacks[28, 9].”
Testing Robustness Against Unforeseen Adversaries (2019, background): “Our results add to a growing line of evidence that evaluating against a single known attack type provides a misleading picture of the robustness of a model [35, 5, 9, 10, 8].”
On Adaptive Attacks to Adversarial Example Defenses (2020, background): “…at an (cid:96) 0 robustness of 55 on MNIST; however, it has been shown that an (cid:96) 0 distortion of 25 is suﬃcient to actually cause humans to change their classiﬁcation of MNIST images (Jacobsen et al., 2019): it is therefore improbable that any defense could achieve a stronger result.”