Adversarial examples in the physical world
By Alexey Kurakin, Ian J. Goodfellow, Samy Bengio. 2016
6,799 citations recorded by the source corpus.
Most existing machine learning classifiers are highly vulnerable to adversarial examples. An adversarial example is a sample of input data which has been modified very slightly in a way that is intended to cause a machine learning classifier to misclassify it. In many cases, these modifications can be so subtle that a human observer does not even notice the modification at all, yet the classifier still makes a mistake. Adversarial examples pose security concerns because they could be used to perform an attack on machine learning systems, even if the adversary has no access to the underlying model. Up to now, all previous work have assumed a threat model in which the adversary can feed data directly into the machine learning classifier. This is not always the case for systems operating in the physical world, for example those which are using signals from cameras and other sensors as an input. This paper shows that even in such physical world scenarios, machine learning systems are vulnerable to adversarial examples. We demonstrate this by feeding adversarial images obtained from cell-phone camera to an ImageNet Inception classifier and measuring the classification accuracy of the system. We find that a large fraction of adversarial examples are classified incorrectly even when perceived through the camera.
Subfields: Artificial Intelligence, Hardware and Architecture, Safety, Robustness & Interpretability, Signal Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Percy Liang
- Mohit Bansal
- Quoc V. Le
- Sergey Levine
- Jianfeng Gao
- Bernt Schiele
- Ruslan Salakhutdinov
In Hardware and Architecture
In Safety, Robustness & Interpretability
- Nicholas Carlini
- Matthias Hein
- Nicolas Papernot
- Bo Li
- Cho-Jui Hsieh
- Huan Zhang
- Dawn Song
- Bernt Schiele
In Signal Processing
- Nicholas Carlini
- Bo Li
- Yoshua Bengio
- Nicolas Papernot
- Bernt Schiele
- Matthias Hein
- Aleksander Madry
- Dawn Song
Citations in the corpus
- Natural Adversarial Examples (2019, background): “Other work has shown how to print 2D [34, 7] or 3D [41, 2] objects that fool classifiers.”
- A Closer Look at Memorization in Deep Networks (2017, background): “To achieve this goal, we build on the intuition that the number of different decision regions into which an input space is partitioned reflects the complexity of the learned hypothesis (Sokolic et al., 2016).”
- Why do deep convolutional networks generalize so poorly to small image transformations? (2018, background): “Despite the excellent performance of CNNs on object recognition, the vulnerability to adversarial attacks suggests that superficial changes can result in highly non-human shifts in prediction [43, 35, 26, 1, 15, 41, 3, 40].”
- How To Backdoor Federated Learning (2018, background): “Adversarial examples [21, 37, 52] are deliberately crafted to be misclassified by the model.”
- Certified Robustness to Adversarial Examples with Differential Privacy (2018, methodology): “We also implemented variants of iterative Projected Gradient Descent (PGD) attack as described in [37], modified to average gradients over 15 noise draws per step, and performing each attack 15 times with a small random jump as initialization.”
- Deep Variational Information Bottleneck (2017, background): “…on image pixels to find a minimal perturbation that changes the model’s classification; single-step gradient-based attacks (Goodfellow et al., 2015; Kurakin et al., 2017; Huang et al., 2015), which choose a gradient direction of the image pixels at some loss and then take a single step in that…”
- Adversarial Training for Free! (2019, methodology): “…the Fast Gradient Sign Method (FGSM) by Goodfellow et al. [2015] is one of the most popular non-targeted methods that uses the sign of the gradients to construct an adversarial example in one iteration: The Basic Iterative Method (BIM) by Kurakin et al. [2016a] is an iterative version of FGSM.”
- Wasserstein Adversarial Examples via Projected Sinkhorn Iterations (2019, methodology): “If we consider the (cid:96) ∞ ball B ∞ ( x, (cid:15) ) = { x + ∆ : (cid:107) ∆ (cid:107) ∞ ≤ (cid:15) } and use steepest descent with respect to the (cid:96) ∞ norm, then we recover the Basic Iterative Method originally presented by Kurakin et al. (2017).”
- Feature Denoising for Improving Adversarial Robustness (2018, background): “4We have also evaluated other attackers, including FGSM [6], iterative FGSM [12], and its momentum variant [3].”
- SentiNet: Detecting Localized Universal Attacks Against Deep Learning Systems (2018, methodology): “Here, we focus on an adversary that causes misclassifications to uncompromised models by crafting ad-hoc malicious objects, e.g., adversarial patches [26], [20], [3], [9], [10], [2].”
- Stronger data poisoning attacks break data sanitization defenses (2018, background): “This line of research was sparked by the striking discovery that test images could be perturbed in a visually-imperceptible way and yet fool state-of-the-art neural network image classifiers (Szegedy et al., 2014; Goodfellow et al. 2015; Carlini et al., 2016; Kurakin et al., 2016; Papernot et al., 2”
- Spectral Signatures in Backdoor Attacks (2018, background): “We also point out that backdoor poisoning is related to adversarial examples [13, 26, 21, 12, 29, 4, 24, 32].”
- Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples (2018, methodology): “However, we note that (1) adversarial retraining has been shown to be difficult at ImageNet scale (Kurakin et al., 2016b Given an image x , for each pixel color x i,j,c , the l -level ther-mometer encoding τ ( x i,j,c ) is a l -dimensional vector where τ ( x i,j,c ) k = 1 if if x i,j,c > k/l , and 0…”
- Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning (2017, background): “In particular, a body of literature studying adversarial examples belongs to this category [29], [53], [36], [48], [54], [52], [15], [40], [25].”
- Decision-Based Adversarial Attacks: Reliable Attacks Against Black-Box Machine Learning Models (2017, methodology): “Examples are the Fast-Gradient Sign Method (FGSM), the Basic Iterative Method (BIM) (Kurakin et al., 2016), DeepFool (MoosaviDezfooli et al.”
- ZOO: Zeroth Order Optimization Based Black-box Attacks to Deep Neural Networks without Training Substitute Models (2017, background): “baseline algorithm for improved adversarial example generation [21, 22], and it can be viewed as an attack framework based on first-order projected gradient descent [27].”
- Certified Defenses for Data Poisoning Attacks (2017, background): “…striking recent security vulnerability discovered in machine learning systems is adversarial test images that can fool image classifiers despite being imperceptible from normal images (Szegedy et al., 2014; Goodfellow et al., 2015; Carlini et al., 2016; Kurakin et al., 2016; Papernot et al., 2016a).”
- Ensemble Adversarial Training: Attacks and Defenses (2017, methodology): “The I-FGSM usually induces higher error rates than the FGSM (for fixed ) [7, 8].”