Recovery RL: Safe Reinforcement Learning With Learned Recovery Zones

By Brijen Thananjeyan, Ashwin Balakrishna, Suraj Nair, Michael Luo, Krishnan Srinivasan, Minho Hwang, Joseph E. Gonzalez, Julian Ibarz, Chelsea Finn, Ken Goldberg. IEEE Robotics and Automation Letters, 2020

307 citations recorded by the source corpus.

Safety remains a central obstacle preventing widespread use of RL in the real world: learning new tasks in uncertain environments requires extensive exploration, but safety requires limiting exploration. We propose Recovery RL, an algorithm which navigates this tradeoff by (1) leveraging offline data to learn about constraint violating zones before policy learning and (2) separating the goals of improving task performance and constraint satisfaction across two policies: a task policy that only optimizes the task reward and a recovery policy that guides the agent to safety when constraint violation is likely. We evaluate Recovery RL on 6 simulation domains, including two contact-rich manipulation tasks and an image-based navigation task, and an image-based obstacle avoidance task on a physical robot. We compare Recovery RL to 5 prior safe RL methods which jointly optimize for task performance and safety via constrained optimization or reward shaping and find that Recovery RL outperforms the next best prior method across all domains. Results suggest that Recovery RL trades off constraint violations and task successes 2–20 times more efficiently in simulation domains and 3 times more efficiently in physical experiments. See https://tinyurl.com/rl-recovery for videos and supplementary material.

Subfields: Artificial Intelligence, Automotive Engineering, Control and Systems Engineering

View Paper: https://www.semanticscholar.org/paper/431dc05ac25510de6264084434254cca877f9ab3

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Automotive Engineering

In Control and Systems Engineering

Citations in the corpus

Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learning (2021, background): “The works (85, 86, 87) make use of such a critic and then resort to various fallback schemes to determine a safer alternative input.”
Safety Alignment Should Be Made More Than Just a Few Tokens Deep (2024, unknown): “Our data augmentation approach in Section 3 relates to exploration requirements for optimal learning via policy gradient methods (Agarwal et al., 2021), learning recovery policies (Thananjeyan et al., 2021), and safe control theory (Burridge et al., 1999; Ames et al., 2019).”
How to train your robot with deep reinforcement learning: lessons we have learned (2021, background): “The task policy, the recovering policy and the classiﬁer for safety can all be learned simultaneously (Eysenbach et al. 2018; Thananjeyan et al. 2020).”
Policy Gradient Bayesian Robust Optimization for Imitation Learning (2021, background): “…to a single task rewards (Heger, 1994; Shen et al., 2014; Tamar et al., 2014; Tang et al., 2019), establish convergence to safe regions of the MDP (Thananjeyan et al., 2020b;a), or optimize a policy to avoid constraint violations (Achiam et al., 2017; Fisac et al., 2018; Thananjeyan et al., 2021).”
VIMA: General Robot Manipulation with Multimodal Prompts (2022, unknown): “, 2021; Lynch & Sermanet, 2021), constraint satisfaction (Bharadhwaj et al., 2021; Srinivasan et al., 2020; Thananjeyan et al., 2021), one-shot imitation (Paine et al.”
Autonomous Reinforcement Learning via Subgoal Curricula (2021, background): “This setting can likely be addressed by leveraging ideas from the literature in safe reinforcement learning [11, 42, 3, 5].”