Tactics of Adversarial Attack on Deep Reinforcement Learning Agents

By Yen-Chen Lin, Zhang-Wei Hong, Yuan-Hong Liao, Meng-Li Shih, Ming-Yu Liu, Min Sun. 2017

474 citations recorded by the source corpus.

We introduce two tactics, namely the strategically-timed attack and the enchanting attack, to attack reinforcement learning agents trained by deep reinforcement learning algorithms using adversarial examples. In the strategically-timed attack, the adversary aims at minimizing the agent's reward by only attacking the agent at a small subset of time steps in an episode. Limiting the attack activity to this subset helps prevent detection of the attack by the agent. We propose a novel method to determine when an adversarial example should be crafted and applied. In the enchanting attack, the adversary aims at luring the agent to a designated target state. This is achieved by combining a generative model and a planning algorithm: while the generative model predicts the future states, the planning algorithm generates a preferred sequence of actions for luring the agent. A sequence of adversarial examples is then crafted to lure the agent to take the preferred sequence of actions. We apply the proposed tactics to the agents trained by the state-of-the-art deep reinforcement learning algorithm including DQN and A3C. In 5 Atari games, our strategically-timed attack reduces as much reward as the uniform attack (i.e., attacking at every time step) does by attacking the agent 4 times less often. Our enchanting attack lures the agent toward designated target states with a more than 70% success rate. Example videos are available at http://yclin.me/adversarial_attack_RL/.

Subfields: Artificial Intelligence, Reinforcement Learning, Safety, Robustness & Interpretability, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

In Safety, Robustness & Interpretability

In Signal Processing

Citations in the corpus

Scaling Laws for Reward Model Overoptimization (2022, methodology): “Many works have demonstrated the existence of adversarial examples in all kinds of neural networks [Szegedy et al., 2013, Lin et al., 2017, Ebrahimi et al., 2018, Dai et al., 2018], and proposed methods to measure and increase neural network robustness [Gu and Rigazio, 2014, Zheng et al., 2016,…”
Certified Defenses for Data Poisoning Attacks (2017, background): “Recent adversarial attacks on reinforcement learners (Huang et al., 2017; Behzadan and Munir, 2017; Lin et al., 2017) blend train and test vulnerabilities.”
Recent Progress on Generative Adversarial Networks (GANs): A Survey (2019, background): “Currently, there are already some works of attack on Convolutional Neural Network (CNN) [72]–[74], Recurrent Neural Networks (RNN) [75] and Deep Reinforcement Learning (DeepRL) [76], [77] and GANs [78].”
The Space of Transferable Adversarial Examples (2017, background): “Adversarial examples can be used to craft human-recognizable images that are misclassified by computer vision models [23, 8, 12, 16], software containing malware but classified as benign [13, 26, 9, 10], and game environments that force reinforcement learning agents to misbehave [11, 4, 15].”
Adversarial Example Defenses: Ensembles of Weak Defenses are not Strong (2017, background): “Research on other applications of neural networks has also encountered adversarial examples on different tasks beyond image classification, including deep policies in reinforcement learning and generative models [14, 13, 12, 2, 18].”
Threat of Adversarial Attacks on Deep Learning in Computer Vision: A Survey (2018, background): “[133] proposed two different adversarial attacks for the agents trained by deep reinforcement learning [155].”
Disentangling Adversarial Robustness and Generalization (2018, background): “Finally, adversarial examples have been applied to a wide variety of tasks, also beyond computer vision, e.g., [26, 18, 93, 47, 35, 54, 3, 16].”
Adversarial Attacks on Stochastic Bandits (2018, background): “The learned policy may be lured to visit certain target states when adversarial examples are driven [19], or have inferior generalization ability when training examples are corrupted [14].”
Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations (2020, background): “, [27, 36]) and cannot be characterized by existing tools such as partially observable Markov decision process (POMDP), because the conditional observation probabilities in POMDP cannot capture the adversarial (worst case) scenario.”
Single and Multi-Agent Deep Reinforcement Learning for AI-Enabled Wireless Networks: A Tutorial (2020, methodology): “To completely trust DRL-based methods in real-world critical applications, an understanding of the vulnerabilities of these methods and addressing them is a central concern in the deployment of AI-empowered systems [130], [131].”
Keeping the Bad Guys Out: Protecting and Vaccinating Deep Learning with JPEG Compression (2017, methodology): “This discovery has gained tremendous interest, and many new attack algorithms have been invented [4, 18, 17, 21] and applied to other domains such as malware detection [5, 7], sentiment analysis [22], and reinforcement learning [14, 8].”
Adversarial Policies: Attacking Deep Reinforcement Learning (2019, background): “Recent work by Lin et al. (2017) generates a sequence of perturbations guiding the victim to a target state.”
Can you fool AI with adversarial examples on a visual Turing test? (2017, background): “While most previous works focus on image classification models, a new line of research studies adversarial examples against deep neural networks for other tasks, such as recurrent neural networks for text processing [43], [20], deep reinforcement learning models for game playing [29], [19], [24], se”
Robust Deep Reinforcement Learning with Adversarial Attacks (2017, methodology): “Lin et al. [2017] proposed attack that is similar in spirit to Carlini and Wagner [2016] attack on images.”
VALUATION : A N ADVERSARIAL APPROACH TO UNCOVER CATASTROPHIC FAILURES (2018, unknown): “Unlike previous work on adversarial examples in reinforcement learning (Huang et al., 2017; Lin et al., 2017), we do not allow adversaries to generate inputs outside the distribution on which the agent is trained.”
Fooling Vision and Language Models Despite Localization and Attention Mechanism (2017, background): “Another line of research studies adversarial examples against deep neural networks for other tasks, such as recurrent neural networks for text processing [59, 28], deep reinforcement learning models for game playing [41, 26, 34], semantic segmentation [14, 70], and object detection [24, 70].”
Adversarial Attacks and Defenses in Deep Learning (2020, background): “Lin et al. [56] propose two attack tactics for deep reinforcement learning, namely the strategically timed attack and the enchanting attack.”
Attacks on Machine Learning: Lurking Danger for Accountability (2019, methodology): “Attack [4] Linear Regression Supervised Yes/No Poisoning Attack [8, 35, 41] No Model Inversion [27] Lowd-Meek Attack [44, 72] Logistic Regression Supervised No Equation-solving Attack [49] Hyperparameter Stealing [73] Adversarial Examples [52, 70, 71] Multi-class Logistic Regression Supervised No Eq”