Robust Adversarial Reinforcement Learning

By Lerrel Pinto, James Davidson, Rahul Sukthankar, Abhinav Kumar Gupta. 2017

1,078 citations recorded by the source corpus.

Deep neural networks coupled with fast simulation and improved computation have led to recent successes in the field of reinforcement learning (RL). However, most current RL-based approaches fail to generalize since: (a) the gap between simulation and real world is so large that policy-learning approaches fail to transfer; (b) even if policy learning is done in real world, the data scarcity leads to failed generalization from training to test scenarios (e.g., due to different friction or object masses). Inspired from H∞ control methods, we note that both modeling errors and differences in training and test scenarios can be viewed as extra forces/disturbances in the system. This paper proposes the idea of robust adversarial reinforcement learning (RARL), where we train an agent to operate in the presence of a destabilizing adversary that applies disturbance forces to the system. The jointly trained adversary is reinforced - that is, it learns an optimal destabilization policy. We formulate the policy learning as a zero-sum, minimax objective function. Extensive experiments in multiple environments (InvertedPendulum, HalfCheetah, Swimmer, Hopper, Walker2d and Ant) conclusively demonstrate that our method (a) improves training stability; (b) is robust to differences in training/test conditions; and c) outperform the baseline even in the absence of the adversary.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

CausalWorld: A Robotic Manipulation Benchmark for Causal Structure and Transfer Learning (2020, methodology): “…were used in experiments in the generalization for RL literature, which have played an important role in advancing the ﬁeld (Packer et al., 2018; Rajeswaran et al., 2017; Pinto et al., 2017; Yu et al., 2017; Henderson et al., 2017a; Dulac-Arnold et al., 2020; Chevalier-Boisvert et al., 2018).”
Solving Rubik's Cube with a Robot Hand (2019, methodology): “[83] propose to use adversarial training to obtain more robust policies and show that it also helps with transfer to physical robots [82].”
Sim-to-Real: Learning Agile Locomotion For Quadruped Robots (2018, methodology): “Another popular method to improve the robustness of a learned controller is to add random perturbations [37].”
Learning dexterous in-hand manipulation (2018, background): “[48] propose to use adversarial training to obtain more robust policies and show that it also helps with transfer to physical robots [47].”
Sim-to-Real Transfer of Robotic Control with Dynamics Randomization (2017, background): “robust policies through adversarial training schemes [22], [23].”
Adversarial Risk and the Dangers of Evaluating Against Weak Attacks (2018, background): “…examples are the most straightforward to study in the visual domain, they have also been demonstrated in other domains such as reinforcement learning (Huang et al., 2017), robotics (Pinto et al., 2017), malware detection (Grosse et al., 2016), and natural language processing (Jia and Liang, 2017).”
A survey and critique of multiagent deep reinforcement learning (2018, methodology): “They address this issue by taking ideas from robust reinforcement learning [257] which implicitly adopts the minimax idea by using the worst noise concept [258].”
Residual Reinforcement Learning for Robot Control (2018, methodology): “Still, RL has been used to allow robots to learn tasks such as playing table tennis [6], swinging up a cartpole and balancing a unicycle [7], grasping [8], [9], opening a door [10], and general manipulation tasks [11], [12].”
Reinforcement Learning for Selective Key Applications in Power Systems: Recent Advances and Future Challenges (2021, methodology): “• Adversarial RL [97], [178] adopts a two-player game structure where the learner agent learns to take actions against an adversarial agent whose objective is different from or even opposite to the learner agent.”
Robust Deep Reinforcement Learning with Adversarial Attacks (2017, methodology): “Morimoto and Doya [2005] and Pinto et al. [2017] took an indirect approach where instead of sampling worst trajectory from rollout, they employ an adversary which applies control action and tries to push the RL agent into possible bad states.”
A Survey on Vision--Language--Action Models for Embodied AI (2024, positive): “Robust adversarial reinforcement learning (RARL) [306] incorporates adversarial agents to enhance generalization.”
Learning scheduling algorithms for data processing clusters (2018, background): “To increase robustness of a scheduling policy against such changes, it may be helpful to train the agent on worst-case situations or adversarial workloads, drawing on the emerging literature on robust adversarial RL [64].”
The Ingredients of Real-World Robotic Reinforcement Learning (2020, methodology): “This has proven to be difﬁcult for policy gradient style algorithms (Pinto et al., 2017a) due to challenging representation learning problems.”
Applications of reinforcement learning in energy systems (2021, neutral): “[174], for example [175], and [176].”
How to train your robot with deep reinforcement learning: lessons we have learned (2021, methodology): “We use domain randomization, which samples different physical parameters, or add perturbation forces, either randomly or adversarially (Pinto et al. 2017), to the robot during training, to force it to learn to react under a wide variety of observations.”
Failure-Scenario Maker for Rule-Based Agent using Multi-agent Adversarial Reinforcement Learning and its Application to Autonomous Driving (2019, methodology): “[Pinto et al. , 2017] consider a two-player zero-sum game in which the pro-tagonist gets a reward r while the adversary gets a reward − r .”