The Space of Transferable Adversarial Examples

By Florian Tramèr, Nicolas Papernot, Ian J. Goodfellow, Dan Boneh, Patrick Drew McDaniel. 2017

602 citations recorded by the source corpus.

Adversarial examples are maliciously perturbed inputs designed to mislead machine learning (ML) models at test-time. They often transfer: the same adversarial example fools more than one model. In this work, we propose novel methods for estimating the previously unknown dimensionality of the space of adversarial inputs. We find that adversarial examples span a contiguous subspace of large (~25) dimensionality. Adversarial subspaces with higher dimensionality are more likely to intersect. We find that for two different models, a significant fraction of their subspaces is shared, thus enabling transferability. In the first quantitative analysis of the similarity of different models' decision boundaries, we show that these boundaries are actually close in arbitrary directions, whether adversarial or benign. We conclude by formally studying the limits of transferability. We derive (1) sufficient conditions on the data distribution that imply transferability for simple model classes and (2) examples of scenarios in which transfer does not occur. These findings indicate that it may be possible to design defenses against transfer-based attacks, even for models that are vulnerable to direct attacks.

Subfields: Artificial Intelligence, Hardware and Architecture, Safety, Robustness & Interpretability, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Hardware and Architecture

In Safety, Robustness & Interpretability

In Signal Processing

Citations in the corpus

Online Adversarial Attacks (2021, background): “Moreover, it has been observed that, in practice, adversarial examples transfer across models (Papernot et al., 2016; Tramèr et al., 2017).”
Towards Deep Learning Models Resistant to Adversarial Attacks (2017, background): “A more recent paper [25] also explores the transferability phenomenon.”
Ensemble Adversarial Training: Attacks and Defenses (2017, background): “[20] find that adversarial training fails to sufficiently displace a model’s decision boundary to prevent adversarial examples from transferring.”
AI2: Safety and Robustness Certification of Neural Networks with Abstract Interpretation (2018, neutral): “, [12], [40], [43]).”
Stochastic Activation Pruning for Robust Adversarial Defense (2018, methodology): “To defend against adversarial examples some papers propose training the neural network on adversarial examples themselves, either using the same model (Goodfellow et al., 2014; Madry et al., 2017), or using an ensemble of models (Tramèr et al., 2017a).”
Generating Adversarial Examples with Adversarial Networks (2018, methodology): “(Goodfellow et al., 2015), ensemble adversarial training (Ensemble) (Tramèr et al., 2017b), and iterative training (Iter.”
The Robust Manifold Defense: Adversarial Training using Generative Models (2017, background): “46, corresponding to an average squared diﬀerence of 2 · 10 − 4 per pixel-channel.”
PixelDefend: Leveraging Generative Models to Understand and Defend against Adversarial Examples (2017, background): “However, recent work (Papernot et al., 2016a; Liu et al., 2016; Tramèr et al., 2017) has shown that adversarial images can transfer across different classiﬁers.”
Adversarial Examples, Uncertainty, and Transfer Testing Robustness in Gaussian Process Hybrid Deep Networks (2017, background): “What is worrying about these adversarial examples is that they are often transferable between di�erent architectures [57, 38, 40, 50, 20] and even between di�erent machine learning methods [47, 59].”
Improving the Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing their Input Gradients (2017, background): “Although there has recently been a great deal of research in adversarial defenses, many of these methods have struggled to achieve robustness to transferred adversarial examples (Tramèr et al. 2017b).”
SoK: Security and Privacy in Machine Learning (2018, methodology): “introduced the Gradient Aligned Adversarial Subspace method, which uses a firstorder approximation similar to the one used to define the fast gradient sign method to directly estimate the dimensionality of the dense space formed by adversarial examples [67].”
Threat of Adversarial Attacks on Deep Learning in Computer Vision: A Survey (2018, methodology): “[131] proposed a method to estimate the dimensionality of the space of the adversarial examples.”
Characterizing Adversarial Subspaces Using Local Intrinsic Dimensionality (2018, methodology): “Similarly, it has also been found that the boundaries of adversarial subspaces are close to legitimate data points in adversarial directions, and that the higher the number of orthogonal adversarial directions of these subspaces, the more transferable they are to other models (Tramèr et al., 2017).”
Adversarial Attacks on Neural Networks for Graph Data (2018, background): “Even more, the adversarial examples generated for one model are often also harmful when using another model: known as transferability [33].”