Adversarial Manipulation of Deep Representations

By Sara Sabour, Yanshuai Cao, Fartash Faghri, David J. Fleet. 2015

297 citations recorded by the source corpus.

We show that the representation of an image in a deep neural network (DNN) can be manipulated to mimic those of other natural images, with only minor, imperceptible perturbations to the original image. Previous methods for generating adversarial images focused on image perturbations designed to produce erroneous class labels, while we concentrate on the internal layers of DNN representations. In this way our new class of adversarial images differs qualitatively from others. While the adversary is perceptually similar to one image, its internal representation appears remarkably similar to a different image, one from a different class, bearing little if any apparent similarity to the input; they appear generic and consistent with the space of natural images. This phenomenon raises questions about DNN representations, as well as the properties of natural images themselves.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Safety, Robustness & Interpretability

Citations in the corpus

Deep Variational Information Bottleneck (2017, methodology): “…attacks (Szegedy et al., 2014; Carlini & Wagner, 2016; Moosavi-Dezfooli et al., 2016; Papernot et al., 2015; Robinson & Graham, 2015; Sabour et al., 2016), which directly run an optimizer such as L-BFGS or ADAM (Kingma & Ba, 2015) on image pixels to ﬁnd a minimal perturbation that…”
Fundamental Tradeoffs between Invariance and Sensitivity to Adversarial Perturbations (2020, background): “ect for models’ internal representations. Denoting the logit layer of a model as z(x), they show that for robust models it is hard to ﬁnd inputs x;x such that O(x) 6= O(x) and z(x) ˇz(x). Conversely, Sabour et al. (2016); Jacobsen et al. (2019) show that excessive invariance of feature layers is com”
Exploiting Excessive Invariance caused by Norm-Bounded Adversarial Robustness (2019, methodology): “Furthermore, the above definition incorporates also adversarial perturbations designed for hidden features as in (Sabour et al., 2016), while usually the decision of the classifier D (argmax-operation on logits) is used as the perturbation target.”
AI2: Safety and Robustness Certification of Neural Networks with Abstract Interpretation (2018, background): “For example, [30] showed how to find adversarial examples without starting from a test point, [41] generated adversarial examples using random perturbations, [35] demonstrated that even intermediate layers are not robust, and [14] generated adversarial examples for malware classification.”
Robust Physical-World Attacks on Deep Learning Models (2017, background): “Recent work in computer vision, however, has demonstrated that DNNs are vulnerable to adversarial perturbations [3], [5], [8], [12], [13], [22], [23], [28].”
NO Need to Worry about Adversarial Examples in Object Detection in Autonomous Vehicles (2017, background): “In the past few years, researchers tried to explain why neural nets are susceptible to such examples despite their impressive success on random test datasets [6, 5, 2], suggested new methods to generate adversarial examples and measure NN robustness against them [4, 21, 16, 9, 17, 19, 13], and propo”
Adversarial Examples for Generative Models (2017, methodology): “An interesting variation of this type of attack can be found in [18].”
Adversarial Examples Detection in Deep Networks with Convolutional Filter Statistics (2016, methodology): “[23] proposes an approach to generate adversarials that match the convolutional filter outputs as well as perturbing the data.”
Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning (2018, methodology): “However, phenomena like adversarial examples— especially those produced by feature adversaries [33], which we cover extensively in Section VII-C—or the lack of invariance to translations [78] indicate that representations learned by DNNs are not as robust as the technical community initially expecte”
Adversarial Examples: Attacks and Defenses for Deep Learning (2017, background): “[67] performed a targeted attack by minimizing the distance of the representation of internal neural network layers instead of the output layer.”
Universal Adversarial Perturbations (2016, background): “The robustness of image classiﬁers to structured and un-structured perturbations have recently attracted a lot of attention [19, 16, 20, 3, 4, 12, 13, 14].”
Measuring Neural Net Robustness with Constraints (2016, neutral): “Given a parameter , the adversarial frequency φ(f, ) def = Prx∗∼D[ρ(f,x∗) ≤ ] measures how often f fails to be (x∗, )-robust.”
Robustness of classifiers: from adversarial to random noise (2016, background): “More recently, following the recent results on the instability of deep neural networks to worst-case perturbations [17], several works have provided explanations of the phenomenon [3, 5, 14, 18], and designed more robust networks [6, 8, 20, 13, 15, 12].”
Threat of Adversarial Attacks on Deep Learning in Computer Vision: A Survey (2018, background): “[26] showed the possibility of generating adversarial examples by altering the internal layers of deep neural networks.”
Technical Report on the CleverHans v2.1.0 Adversarial Examples Library (2016, background): “Feature Adversaries were introduced by Sabour et al. [20].”
Adversarial Diversity and Hard Positive Generation (2016, methodology): “Sabour et al. concluded that Lp measures are not well matched to human perception [16].”
EagleEye: Attack-Agnostic Defense against Adversarial Inputs (Technical Report) (2018, background): “Addressing the vulnerabilities of deep learning systems to adversarial inputs is more challenging for they are designed to model highly nonlinear, nonconvex functions [31, 36].”
The vulnerability of learning to adversarial perturbation increases with intrinsic dimensionality (2017, background): “Moreover, it has recently been shown that even models with parameters picked at random are unstable with respect to adversarial perturbation [3].”