A Boundary Tilting Persepective on the Phenomenon of Adversarial Examples

By Thomas Tanay, Lewis D. Griffin. 2016

289 citations recorded by the source corpus.

Deep neural networks have been shown to suffer from a surprising weakness: their classification outputs can be changed by small, non-random perturbations of their inputs. This adversarial example phenomenon has been explained as originating from deep networks being "too linear" (Goodfellow et al., 2014). We show here that the linear explanation of adversarial examples presents a number of limitations: the formal argument is not convincing, linear classifiers do not always suffer from the phenomenon, and when they do their adversarial examples are different from the ones affecting deep networks. We propose a new perspective on the phenomenon. We argue that adversarial examples exist when the classification boundary lies close to the submanifold of sampled data, and present a mathematical analysis of this new perspective in the linear case. We define the notion of adversarial strength and show that it can be reduced to the deviation angle between the classifier considered and the nearest centroid classifier. Then, we show that the adversarial strength can be made arbitrarily high independently of the classification performance due to a mechanism that we call boundary tilting. This result leads us to defining a new taxonomy of adversarial examples. Finally, we show that the adversarial strength observed in practice is directly dependent on the level of regularisation used and the strongest adversarial examples, symptomatic of overfitting, can be avoided by using a proper level of regularisation.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

Adversarial Examples Are Not Bugs, They Are Features (2019, methodology): “From this point of view, it is natural to treat adversarial robustness as a goal that can be disentangled and pursued independently from maximizing accuracy [Mad+18; SHS19; Sug+19], either through improved standard regularization methods [TG16] or pre/post-processing of network inputs/outputs [Ues+1”
Feature Purification: How Adversarial Training Performs Robust Deep Learning (2020, background): “In particular, we found that the following fundamental questions remain largely unaddressed: Questions Why do adversarial examples exist when we train the neural networks using the original training data set? How can adversarial training further “robustify” the trained neural networks against these ”
On Detecting Adversarial Perturbations (2017, background): “Adopting the boundary tilting perspective of Tanay & Grifﬁn (2016), strong adversarial examples occur in situations in which classiﬁcation boundaries are tilted against the data manifold such that they lie close and nearly parallel to the data manifold.”
Is Deep Learning Safe for Robot Vision? Adversarial Examples Against the iCub Humanoid (2017, background): “Note that this is a general issue for deep networks, not only specific to ImageNet [26, 9, 25, 10, 19, 20].”
Detecting Adversarial Samples from Artifacts (2017, background): “Although this explanation has been the most well-accepted in the field, it was recently weakened by counterexamples (Tanay & Griffin, 2016).”
Threat of Adversarial Attacks on Deep Learning in Computer Vision: A Survey (2018, background): “In the opinion of Tanay and Griffin, the linearity hypothesis [23] about the existence of adversarial examples is “unconvincing”.”
Adversarial vulnerability for any classifier (2018, background): “Other works have instead studied the existence of adversarial perturbations, under strong assumptions on the data distribution [18, 21].”
Characterizing Adversarial Subspaces Using Local Intrinsic Dimensionality (2018, background): “As pointed out by (Goodfellow et al., 2014; Warde-Farley et al., 2016; Tanay & Grifﬁn, 2016), x (cid:48) is very likely to lie outside S (but very close to S — in a high-dimensional contiguous space).”
Disentangling Adversarial Robustness and Generalization (2018, background): “First, we showed that regular adversarial examples indeed leave the manifold, as widely assumed in related work [27, 99, 39, 72, 82].”
Analysis of universal adversarial perturbations (2017, background): “classifiers to adversarial perturbations has been specifically studied, leading to theoretical as well empirical explanations of the phenomenon [9], [10], [11], [12].”
Adversarial Examples on Object Recognition (2020, background): “In summary, although Tanay and Griffin [163] claim to refute the linearity hypothesis, there is still not enough empirical evidence to completely reject it.”
With Friends Like These, Who Needs Adversaries? (2018, background): “Tanay and Griffin [17] begin by taking issue with the above via illustrative toy problems.”
Robustness via Curvature Regularization, and Vice Versa (2018, background): “On the analysis front, many works have analyzed the existence of adversarial examples, and proposed several hypotheses for their existence [6, 8, 22, 5, 12].”
Advances in adversarial attacks and defenses in computer vision: A survey (2021, background): “[30], another popular concept related to the existence of adversarial examples is ‘manifold assumption’ [296], which argues that adversarial examples tend to leave the clean data manifold.”
Defense Against Adversarial Attacks Using Feature Scattering-based Adversarial Training (2019, background): “Natural images live on a low-dimensional manifold, with the training and testing images as samples from it [26, 51, 44, 56].”