Motivating the Rules of the Game for Adversarial Example Research

By Justin Gilmer, Ryan P. Adams, Ian J. Goodfellow, David G. Andersen, George E. Dahl. 2018

240 citations recorded by the source corpus.

Advances in machine learning have led to broad deployment of systems with impressive performance on important problems. Nonetheless, these systems can be induced to make errors on data that are surprisingly similar to examples the learned system handles correctly. The existence of these errors raises a variety of questions about out-of-sample generalization and whether bad actors might use such examples to abuse deployed systems. As a result of these security concerns, there has been a flurry of recent papers proposing algorithms to defend against such malicious perturbations of correctly handled examples. It is unclear how such misclassifications represent a different kind of security problem than other errors, or even other attacker-produced examples that have no specific relationship to an uncorrupted input. In this paper, we argue that adversarial example defense papers have, to date, mostly considered abstract, toy games that do not relate to any specific security concern. Furthermore, defense papers have not yet precisely described all the abilities and limitations of attackers that would be relevant in practical security. Towards this end, we establish a taxonomy of motivations, constraints, and abilities for more plausible adversaries. Finally, we provide a series of recommendations outlining a path forward for future work to more clearly articulate the threat model and perform more meaningful evaluation.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

In Signal Processing

Citations in the corpus

Pretrained Transformers Improve Out-of-Distribution Robustness (2020, background): “However, adversarial examples are often disconnected from real-world performance concerns (Gilmer et al., 2018).”
Natural Adversarial Examples (2019, background): “classiﬁers. For all that, these existing adversarial attacks are all based on synthesized images or objects, and some have questioned whether they provide a reliable window into real-world robustness [14]. Our examples are closer in spirit to the recent thought experiment of an adversarial photograp”
Benchmarking Neural Network Robustness to Common Corruptions and Perturbations (2019, background): “Gilmer et al. (2018a) suggest modifying the problem of adversarial robustness itself for increased real-world applicability.”
Scalable agent alignment via reward modeling: a research direction (2018, background): “However, it is unclear how to deﬁne adversarial perturbation rigorously in a general way (Brown et al., 2018; Gilmer et al., 2018).”
AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty (2019, background): “Gilmer et al. (2018); Gilmer & Hendrycks (2019) argue that robustness to data shift is a pressing problem which greatly affects the reliability of real-world machine learning systems.”
Unsolved Problems in ML Safety (2021, background): “To study adversarial robustness more broadly [61], researchers could consider attacks that are perceptible [142] or whose speciﬁcations are not known beforehand [97, 112].”
Fundamental Tradeoffs between Invariance and Sensitivity to Adversarial Perturbations (2020, background): “Research on adversarial examples (Szegedy et al., 2013; Biggio et al., 2013) is motivated by a spectrum of questions, ranging from the security of models in adversarial settings (Tramèr et al., 2019), to limitations of learned representations under natural distribution shift (Gilmer et al., 2018).”
Exploiting Excessive Invariance caused by Norm-Bounded Adversarial Robustness (2019, methodology): “To do so, we consider the synthetic adversarial spheres problem of Gilmer et al. (2018b).”
Certified Adversarial Robustness via Randomized Smoothing (2019, background): “Regardless of whether or not imperceptible adversarial perturbations truly pose a unique real-world security threat (Gilmer et al., 2018a), they do present an interesting scientific question: under what conditions is it possible to learn a high-accuracy classifier that is invariant to imperceptible…”
Excessive Invariance Causes Adversarial Vulnerability (2018, methodology): “Example 6 (Mis-aligned classiﬁer on Adversarial Spheres) .”
Theoretically Principled Trade-off between Robustness and Accuracy (2019, background): “, 2013), or unrestricted attacks such as adversarial rotations, translations, and deformations (Brown et al., 2018; Engstrom et al., 2017; Gilmer et al., 2018; Xiao et al., 2018; Alaifari et al., 2019; Zhang et al., 2019a).”
A Fourier Perspective on Model Robustness in Computer Vision (2019, background): “These qualify as simple, few query black box attacks that satisfy the content preserving threat model (Gilmer et al., 2018).”
On Evaluating Adversarial Robustness (2019, background): “This threat model is highly limited and does not perfectly match real-world threats (Engstrom et al., 2017; Gilmer et al., 2018).”
Adversarial Examples Are a Natural Consequence of Test Error in Noise (2019, background): “This suggests that defending against worst-case content-preserving perturbations (Gilmer et al., 2018a) requires removing all errors at a scale comparable to the distance between unrelated pairs of images.”
Improving Robustness Without Sacrificing Accuracy with Patch Gaussian Augmentation (2019, background): “Rather than focusing on adversarial examples that are worst-case bounded perturbations, we focus on a more general set of corruptions [18] that models are likely to encounter in real-world settings: the Common Corruptions benchmark [24].”
Unrestricted Adversarial Examples (2018, methodology): “…to clean inputs [Szegedy et al., 2013]; • applying ordinary spatial transformations to the input (such as translations and rotations on images) [Engstrom et al., 2017]; • and performing simple guess-and-check (trying a variety of inputs that seem likely to be misclassiﬁed) [Gilmer et al., 2018].”
RobustBench: a standardized adversarial robustness benchmark (2020, background): “We note that 90 robustness towards small `p-perturbations is a necessary but not sufficient notion of robustness which 91 has been criticized in the literature [41].”
Rademacher Complexity for Adversarially Robust Generalization (2018, methodology): “Typical methods of generating adversarial examples include adding small perturbations that are imperceptible to humans [51], changing surrounding areas of the main objects in images [21], and even simple rotation and translation [17].”