Delving into adversarial attacks on deep policies

By Jernej Kos, Dawn Song. 2017

258 citations recorded by the source corpus.

Adversarial examples have been shown to exist for a variety of deep learning architectures. Deep reinforcement learning has shown promising results on training agent policies directly on raw inputs such as image pixels. In this paper we present a novel study into adversarial attacks on deep reinforcement learning polices. We compare the effectiveness of the attacks using adversarial examples vs. random noise. We present a novel method for reducing the number of times adversarial examples need to be injected for a successful attack, based on the value function. We further explore how re-training on random noise and FGSM perturbations affects the resilience against adversarial examples.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

Certified Robustness to Adversarial Examples with Differential Privacy (2018, background): “Adversarial examples have also been demonstrated in other domains such as reinforcement learning [33] and generative models [32].”
Parametric Noise Injection: Trainable Randomness to Improve Deep Neural Network Robustness Against Adversarial Attack (2018, background): “Owing to the richer information, white-box attack can often achieve higher attacking success rates in comparison to the black-box counterpart, for various applications [40, 24, 21, 35, 32, 4, 11, 7].”
Stochastic Activation Pruning for Robust Adversarial Defense (2018, background): “Previously, (Behzadan & Munir, 2017; Huang et al., 2017; Kos & Song, 2017) have shown that the reinforcement learning agents can also be easily manipulated by adversarial examples.”
Robust Physical-World Attacks on Deep Learning Models (2017, background): “, by modifying an image representing a real-world scene that a cyber-physical system might perceive [7], [9], a fundamental open question, which we answer in this paper, is whether it is possible to create robust physical adversarial perturbations—small modifications to real-world objects themselves”
Adversarial Examples: Attacks and Defenses for Deep Learning (2017, methodology): “Kos and Song [85] used FGSM to attack A3C algorithm”
Adversarial Example Defenses: Ensembles of Weak Defenses are not Strong (2017, background): “Research on other applications of neural networks has also encountered adversarial examples on different tasks beyond image classification, including deep policies in reinforcement learning and generative models [14, 13, 12, 2, 18].”
Can you fool AI with adversarial examples on a visual Turing test? (2017, background): “While most previous works focus on image classification models, a new line of research studies adversarial examples against deep neural networks for other tasks, such as recurrent neural networks for text processing [43], [20], deep reinforcement learning models for game playing [29], [19], [24], se”
On the Connection between Differential Privacy and Adversarial Robustness in Machine Learning (2018, background): “Adversarial examples have also been demonstrated in other domains such as reinforcement learning [24] and generative models [23].”
Stealthy and Efficient Adversarial Attacks against Deep Reinforcement Learning (2020, methodology): “(Kos and Song 2017) proposed some simple methods towards this goal: instead of compromising every environmental state, an adversary can compute and inject the perturbations every N frames.”
Objective Robustness in Deep Reinforcement Learning (2021, background): “In reinforcement learning, agents have also been shown to be vulnerable to small perturbations on input images [15; 21], as well as vulnerable to adversarial policies: policies that attack the agent simply by taking actions in the same environment and thereby modifying the agent’s observations to ca”
Fooling Vision and Language Models Despite Localization and Attention Mechanism (2017, background): “Another line of research studies adversarial examples against deep neural networks for other tasks, such as recurrent neural networks for text processing [59, 28], deep reinforcement learning models for game playing [41, 26, 34], semantic segmentation [14, 70], and object detection [24, 70].”
Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations (2020, background): “Improving Robustness for State Observations in DRL For discrete action RL tasks, Kos & Song [31] first presented preliminary results of adversarial training on Pong (one of the simplest Atari environments) using weak FGSM attacks on pixel space.”
Better the Devil you Know: An Analysis of Evasion Attacks using Out-of-Distribution Adversarial Examples (2019, methodology): “This paper’s focus is evasion attacks which have been proposed against supervised ML algorithms used for image classification [13, 18, 24, 45, 97, 114], object detection [26, 83, 127], image segmentation [6, 41], speech recognition [19, 133] as well as other tasks [27, 48, 65, 129]; generative model”
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (2023, background): “Black-box access to a model (e.g., via API access) is sufficient for many adversarial policy attack algorithms, though white-box access (enabled for example by open-sourced or leaked model weights) enables even stronger exploits [Kos and Song, 2017, Casper et al., 2022].”
PassGAN: A Deep Learning Approach for Password Guessing (2017, background): “In the areas of privacy and security, research on deep learning has focused on three major thrusts: (1) privacypreserving deep learning [80] and differentially-private deep learning [1]; (2) attacks on deep learning models that are trained on private data [6], [28], [81], [37], [90], [40]; and (3) a”
Learning Visually-Grounded Semantics from Contrastive Adversarial Samples (2018, background): “The adversarial attacks spread over multiple domains including image classiﬁcation (Nguyen et al., 2015), image segmentation and object detection (Xie et al., 2017), and textual reading comprehension (Jia and Liang, 2017), and deep reinforcement learning (Kos and Song, 2017).”
Adversarial Policies: Attacking Deep Reinforcement Learning (2019, background): “Recent work has shown that deep RL policies are also vulnerable to adversarial perturbations of image observations (Huang et al., 2017; Kos and Song, 2017).”
Robust Deep Reinforcement Learning with Adversarial Attacks (2017, methodology): “We brieﬂy review the adversarial attacks on images (Goodfellow et al. [2014]) because they have been extended to image based DRL (Huang et al. [2017], Kos and Song [2017], Lin et al. [2017]).”
Say What I Want: Towards the Dark Side of Neural Dialogue Models (2019, background): “[24] try to find the weakness of an object detection system; in [25, 26], researchers study the robustness of generative models, and novel methods to make adversarial examples for deep reinforcement learning based models are introduced in [27, 28].”