Adversarial Attacks on Neural Network Policies

By Sandy H. Huang, Nicolas Papernot, Ian J. Goodfellow, Yan Duan, Pieter Abbeel. 2017

956 citations recorded by the source corpus.

Machine learning classifiers are known to be vulnerable to inputs maliciously constructed by adversaries to force misclassification. Such adversarial examples have been extensively studied in the context of computer vision applications. In this work, we show adversarial attacks are also effective when targeting neural network policies in reinforcement learning. Specifically, we show existing adversarial example crafting techniques can be used to significantly degrade test-time performance of trained policies. Our threat model considers adversaries capable of introducing small perturbations to the raw input of the policy. We characterize the degree of vulnerability across tasks and training algorithms, for a subclass of adversarial-example attacks in white-box and black-box settings. Regardless of the learned task or training algorithm, we observe a significant drop in performance, even with small adversarial perturbations that do not interfere with human perception. Videos are available at this http URL.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

Scalable agent alignment via reward modeling: a research direction (2018, background): “Second, the agent needs to react robustly to perturbations that may cause it to produce unacceptable outcomes unintentionally (Ortega et al., 2018) such as distributional changes and adversarial inputs (Szegedy et al., 2013; Huang et al., 2017).”
Stochastic Activation Pruning for Robust Adversarial Defense (2018, background): “Previously, (Behzadan & Munir, 2017; Huang et al., 2017; Kos & Song, 2017) have shown that the Reinforcement learning agents can also be easily manipulated by adversarial examples.”
Robust Physical-World Attacks on Deep Learning Models (2017, background): “, by modifying an image representing a real-world scene that a cyber-physical system might perceive [7], [9], a fundamental open question, which we answer in this paper, is whether it is possible to create robust physical adversarial perturbations—small modifications to real-world objects themselves”
Delving into adversarial attacks on deep policies (2017, background): “Concurrently and independently from our work (submission to the same ICLR workshop), Huang et al. (2017) also presented a study into adversarial attacks on DRL policies, showing when an attacker injects small adversarial perturbations into every frame, the learned agent will fail.”
Recent Progress on Generative Adversarial Networks (GANs): A Survey (2019, background): “Currently, there are already some works of attack on Convolutional Neural Network (CNN) [72]–[74], Recurrent Neural Networks (RNN) [75] and Deep Reinforcement Learning (DeepRL) [76], [77] and GANs [78].”
Ensemble Bayesian Decision Making with Redundant Deep Perceptual Control Policies (2018, background): “Even small perturbations to the input can alter the output of Deep NNs [7], [8] and the Deep NNs are easily fooled [9].”
Deep Learning: A Critical Appraisal (2018, neutral): “But (Huang, Papernot, Goodfellow, Duan, & Abbeel, 2017).”
Adversarially Robust Generalization Requires More Data (2018, background): “More recently, researchers have demonstrated the phenomenon under the name of adversarial examples in image classification [21, 51], question answering [28], voice recognition [12, 13, 49, 62], and other domains (for instance, see [2, 4, 14, 22, 25, 26, 32, 60]).”
The Space of Transferable Adversarial Examples (2017, background): “Adversarial examples can be used to craft human-recognizable images that are misclassified by computer vision models [23, 8, 12, 16], software containing malware but classified as benign [13, 26, 9, 10], and game environments that force reinforcement learning agents to misbehave [11, 4, 15].”
Theoretically Principled Trade-off between Robustness and Accuracy (2019, background): “, 2013), adversarial defenses have been an imperative object of study in machine learning (Huang et al., 2017), computer vision (Song et al.”
Adversarial Example Defenses: Ensembles of Weak Defenses are not Strong (2017, background): “Research on other applications of neural networks has also encountered adversarial examples on different tasks beyond image classification, including deep policies in reinforcement learning and generative models [15, 14, 12, 2].”
SoK: Security and Privacy in Machine Learning (2018, background): “Huang et al. demonstrated that a RL agent is vulnerable to adversarial perturbations of its environment after it has been trained [68].”
Threat of Adversarial Attacks on Deep Learning in Computer Vision: A Survey (2018, background): “[62] demonstrated that FGSM [23] can also be used to significantly degrade performance of trained policies in the context of deep reinforcement learning.”
Adversarial Risk and the Dangers of Evaluating Against Weak Attacks (2018, background): “In the case of gradient-based adversaries, a model may exploit gradient masking to bypass the speciﬁc adversary, without achieving true adversarial robustness (Tramèr et al., 2017; Goodfellow and Papernot, 2017).”
A Study on Overfitting in Deep Reinforcement Learning (2018, neutral): “The current situation not only makes it difficult to understand the test behaviors like the vulnerabilities to potential adversarial attacks (Huang et al., 2017), but also renders some results difficult to reproduce or compare (Henderson et al., 2017; Machado et al., 2017).”
AI Safety Gridworlds (2017, background): “Recent research has shown that several machine learning methods exhibit a remarkable fragility to inputs with adversarial perturbations (Szegedy et al., 2013; Goodfellow et al., 2014); these adversarial attacks also affect neural network policies in reinforcement learning (Huang et al., 2017a).”
Tactics of Adversarial Attack on Deep Reinforcement Learning Agents (2017, methodology): “We can convert the computed Q-values of actions into a probability distribution over actions using the softmax function (similar to [Huang et al., 2017]) with the temperature constant T .”
Semidefinite relaxations for certifying robustness to adversarial examples (2018, methodology): “The second category draws techniques from formal verification such as SMT [16, 17, 7, 14], which aim to provide tight certificates for any network using discrete optimization.”