Robust Physical-World Attacks on Deep Learning Models

By Ivan Evtimov, Kevin Eykholt, Earlence Fernandes, Tadayoshi Kohno, Bo Li, Atul Prakash, Amir Rahmati, Dawn Song. 2017

621 citations recorded by the source corpus.

Recent studies show that the state-of-the-art deep neural networks (DNNs) are vulnerable to adversarial examples, resulting from small-magnitude perturbations added to the input. Given that that emerging physical systems are using DNNs in safety-critical situations, adversarial examples could mislead these systems and cause dangerous situations.Therefore, understanding adversarial examples in the physical world is an important step towards developing resilient learning algorithms. We propose a general attack algorithm,Robust Physical Perturbations (RP2), to generate robust visual adversarial perturbations under different physical conditions. Using the real-world case of road sign classification, we show that adversarial examples generated using RP2 achieve high targeted misclassification rates against standard-architecture road sign classifiers in the physical world under various environmental conditions, including viewpoints. Due to the current lack of a standardized testing method, we propose a two-stage evaluation methodology for robust physical adversarial examples consisting of lab and field tests. Using this methodology, we evaluate the efficacy of physical adversarial manipulations on real objects. Witha perturbation in the form of only black and white stickers,we attack a real stop sign, causing targeted misclassification in 100% of the images obtained in lab settings, and in 84.8%of the captured video frames obtained on a moving vehicle(field test) for the target classifier.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

In Signal Processing

Citations in the corpus

Benchmarking Neural Network Robustness to Common Corruptions and Perturbations (2019, background): “…new defenses (Lu et al., 2017; Papernot et al., 2017; Metzen et al., 2017; Hendrycks & Gimpel, 2017a) often quickly succumb to new attacks (Evtimov et al., 2017; Carlini & Wagner, 2017; 2016), with some exceptions for perturbations on small images (Schott et al., 2018; Madry et al., 2018).”
Adversarial Filters of Dataset Biases (2020, neutral): “wild” (Eykholt et al., 2018; Jia & Liang, 2017).”
Why do deep convolutional networks generalize so poorly to small image transformations? (2018, background): “Despite the excellent performance of CNNs on object recognition, the vulnerability to adversarial attacks suggests that superficial changes can result in highly non-human shifts in prediction [43, 35, 26, 1, 15, 41, 3, 40].”
Backdooring Convolutional Neural Networks via Targeted Weight Perturbations (2018, background): “These are images that have been perturbed in some way that cause misclassifications when given as inputs to these networks [7, 21, 23, 25, 29, 22].”
PoTrojan: powerful neural-level trojan designs in deep learning models (2018, background): “In [4], I. Evtimov et al. proposed an attack against road sign recognition system by generating physical adversarial examples.”
The Robust Manifold Defense: Adversarial Training using Generative Models (2017, background): “On the contrary, attacks like the robust physical perturbations of traﬃc signs shown in [16] would never make a reasonable person think this is not a stop sign.”
Towards the first adversarially robust neural network model on MNIST (2018, background): “Similarly, real-world adversarial examples against camera-based algorithms such as the ones deployed in autonomous vehicles are possible [Evtimov et al., 2017].”
Deep Learning: A Critical Appraisal (2018, background): “More recently, there have been real-world stop signs, lightly defaced, that have been mistaken for speed limit signs (Evtimov et al., 2017) and 3d-printed turtles that have been mistake for rifles (Athalye, Engstrom, Ilyas, & Kwok, 2017).”
Robustness May Be at Odds with Accuracy (2018, background): “…a number of approaches both to finding these perturbations, and to training models that are robust to them (Goodfellow et al., 2014b; Nguyen et al., 2015; Moosavi-Dezfooli et al., 2016; Carlini & Wagner, 2016; Sharif et al., 2016; Kurakin et al., 2016a; Evtimov et al., 2017; Athalye et al., 2017).”
Motivating the Rules of the Game for Adversarial Example Research (2018, background): “Even if we view Evtimov et al. [40] as using the standard rules with a relatively large (cid:15) and the l 0 metric, their attack still takes a lot of eﬀort and expertise compared to just covering a larger area of the sign with a picture of another sign, placing a bag over the sign, or knocking it…”
Threat of Adversarial Attacks on Deep Learning in Computer Vision: A Survey (2018, background): “[75] are orthogonal to the results in [66].”
EAD: Elastic-Net Attacks to Deep Neural Networks via Adversarial Examples (2017, background): “Moreover, moving beyond ∗Pin-Yu Chen and Yash Sharma contribute equally to this work. the digital space, researchers have shown that these adversarial examples are still effective in the physical world at fooling DNNs (Kurakin, Goodfellow, and Bengio 2016a; Evtimov et al. 2017).”
Adversarial Patch (2017, methodology): “[4] demonstrated various methods for constructing stop signs that are misclassified by models, either by printing out a large poster that looks like a stop sign, or by placing various stickers on a stop sign.”
Understanding Membership Inferences on Well-Generalized Learning Models (2018, background): “ls. Different attacks against machine learning models have been proposed in recent years. For example, reverse engineering attacks [17, 36] steal model parameters and structures; adversarial learning [13,22,27,35] generates misleading examples that will be misclassiﬁed by the model; model inversion ”