Action Robust Reinforcement Learning and Applications in Continuous Control

By Chen Tessler, Yonathan Efroni, Shie Mannor. International Conference on Machine Learning, 2019

304 citations recorded by the source corpus.

A policy is said to be robust if it maximizes the reward while considering a bad, or even adversarial, model. In this work we formalize two new criteria of robustness to action uncertainty. Specifically, we consider two scenarios in which the agent attempts to perform an action $a$, and (i) with probability $\alpha$, an alternative adversarial action $\bar a$ is taken, or (ii) an adversary adds a perturbation to the selected action in the case of continuous action space. We show that our criteria are related to common forms of uncertainty in robotics domains, such as the occurrence of abrupt forces, and suggest algorithms in the tabular case. Building on the suggested algorithms, we generalize our approach to deep reinforcement learning (DRL) and provide extensive experiments in the various MuJoCo domains. Our experiments show that not only does our approach produce robust policies, but it also improves the performance in the absence of perturbations. This generalization indicates that action-robustness can be thought of as implicit regularization in RL problems.

View Paper: https://www.semanticscholar.org/paper/bcdb21ca1703fc6f62df420626e36d138480a6a1

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Huan Zhang
2. Bo Li
3. Chaowei Xiao
4. Hongge Chen
5. Cho-Jui Hsieh
6. Sung Ju Hwang
7. Jinwoo Shin
8. Duane S. Boning

Citations in the corpus

• Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey (2020, background): “It considers a bad, or even adversarial model and tries to maximize the reward as a optimization problem [24], [25].”
• Applications of reinforcement learning in energy systems (2021, unknown): “[174], for example [175], and [176].”
• Model-augmented Prioritized Experience Replay (2022, methodology): “For instance, various methods for safety (Tessler et al., 2019; Pinto et al., 2017; Chow et al., 2018; Wachi & Sui, 2020; Alshiekh et al., 2018; Cheng et al., 2019) should be proposed in parallel of methods for improving the sample efficiency of RL.”
• Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations (2020, background): “[71] considered adversarial perturbations on the action space.”