Adversarial Policies: Attacking Deep Reinforcement Learning

By Adam Gleave, Michael Dennis, Neel Kant, Cody Wild, Sergey Levine, Stuart J. Russell. International Conference on Learning Representations, 2019

437 citations recorded by the source corpus.

Deep reinforcement learning (RL) policies are known to be vulnerable to adversarial perturbations to their observations, similar to adversarial examples for classifiers. However, an attacker is not usually able to directly modify another agent's observations. This might lead one to wonder: is it possible to attack an RL agent simply by choosing an adversarial policy acting in a multi-agent environment so as to create natural observations that are adversarial? We demonstrate the existence of adversarial policies in zero-sum games between simulated humanoid robots with proprioceptive observations, against state-of-the-art victims trained via self-play to be robust to opponents. The adversarial policies reliably win against the victims but generate seemingly random and uncoordinated behavior. We find that these policies are more successful in high-dimensional environments, and induce substantially different activations in the victim policy network than when the victim plays against a normal opponent. Videos are available at this https URL.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/6ff50528f3d7c72772f8c0e3f8398f9dd8e06575

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Advances in adversarial attacks and defenses in computer vision: A survey (2021, background): “[223] showed the existence of adversarial policies in zero-sum games between robots, especially in high dimensional environments.”
Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design (2020, background): “Competition can drive the emergence of complex skills, even some that the developer did not anticipate [5, 13].”
Multi-agent deep reinforcement learning: a survey (2021, background): “However, Gleave et al. (2020) studied the existence of adversarial policies in competitive games and showed that complex policies could be fooled by comparably easy strategies.”
Objective Robustness in Deep Reinforcement Learning (2021, background): “In reinforcement learning, agents have also been shown to be vulnerable to small perturbations on input images [15; 21], as well as vulnerable to adversarial policies: policies that attack the agent simply by taking actions in the same environment and thereby modifying the agent’s observations to ca”
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (2023, background): “Even “superhuman” policies can fail catastrophically against policies specifically designed to exploit them [Gleave et al., 2020b, Wu et al., 2021b, Wang et al., 2022].”
Single and Multi-Agent Deep Reinforcement Learning for AI-Enabled Wireless Networks: A Tutorial (2020, methodology): “To completely trust DRL-based methods in real-world critical applications, an understanding of the vulnerabilities of these methods and addressing them is a central concern in the deployment of AI-empowered systems [130], [131].”
Finding and Visualizing Weaknesses of Deep Reinforcement Learning Agents (2019, background): “In the next section, we will show how to overcome these difﬁculties.”
Unsolved Problems in ML Safety (2021, background): “Specifically, suppose a neural network is used to define a learned utility function; if some other agent (say another neural network) is tasked with maximizing this utility proxy, it would be incentivized to find and exploit any errors in the learned utility proxy, similar to adversarial examples [1”
Stable-Baselines3: Reliable Reinforcement Learning Implementations (2021, methodology): “This allows researchers to easily use the baseline algorithms and components in their experiments (e.g. Klink et al. (2020); Nair et al. (2019); Gleave et al. (2020)), as well as apply RL to novel tasks and environments, like continual learning when attacking WiFi networks (Margaritelli, 2020) or…”