Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learning

By Benjamin Eysenbach, Shixiang Shane Gu, Julian Ibarz, Sergey Levine. 2017

157 citations recorded by the source corpus.

Deep reinforcement learning algorithms can learn complex behavioral skills, but real-world application of these methods requires a large amount of experience to be collected by the agent. In practical settings, such as robotics, this involves repeatedly attempting a task, resetting the environment between each attempt. However, not all tasks are easily or automatically reversible. In practice, this learning process requires extensive human intervention. In this work, we propose an autonomous method for safe and efficient reinforcement learning that simultaneously learns a forward and reset policy, with the reset policy resetting the environment for a subsequent attempt. By learning a value function for the reset policy, we can automatically determine when the forward policy is about to enter a non-reversible state, providing for uncertainty-aware safety aborts. Our experiments illustrate that proper use of the reset policy can greatly reduce the number of manual resets required to learn a task, can reduce the number of unsafe actions that lead to non-reversible states, and can automatically induce a curriculum.

Subfields: Artificial Intelligence, Control and Systems Engineering, Reinforcement Learning, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Reinforcement Learning

In Safety, Robustness & Interpretability

Citations in the corpus

Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, methodology): “When bootstrap ensembles3 are used to represent the Q-function, as is common in prior work (Osband et al., 2016; Eysenbach et al., 2017; Kumar et al., 2019a; Agarwal et al., 2019), typical choices of ‘Unc’ include variance across Q-value predictions of an ensemble of Q-functions (Kumar et al.,…”
Continual Learning of Control Primitives: Skill Discovery via Reset-Games (2020, methodology): “We hypothesize that is due to the improved state coverage of our approach, as an explicit reset policy (green curve) [13] performs much worse.”
Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization (2020, background): “While there are abundant works on safety for RL (Chow et al., 2015; Eysenbach et al., 2018; Chow et al., 2018; Ray et al., 2019; Chow et al., 2019), they often do not provide guarantees in practice when combined with neural networks and stochastic optimization.”
Reset-Free Lifelong Learning with Skill-Space Planning (2020, background): “The traditional approach to non-episodic RL is to explicitly learn a policy to reset the environment (Even-Dar et al., 2005; Han et al., 2015; Eysenbach et al., 2017).”
Avoiding Side Effects By Considering Future Tasks (2020, background): “An auxiliary reward for avoiding irreversible actions [6] is equivalent to the future task auxiliary reward with only one possible future task (i = 1), where the goal state g1 is the starting state s0.”
How to train your robot with deep reinforcement learning: lessons we have learned (2021, background): “These approaches can use ensemble models to estimate uncertainty (Deisenroth and Rasmussen 2011; Eysenbach et al. 2018) of certain predictions, which can be a good indicator whether any unsafe behavior may happen, or can directly learn the probability of future unsafe behaviors from experience…”
The Ingredients of Real-World Robotic Reinforcement Learning (2020, methodology): “Eysenbach et al. (2018) provide an algorithm to learn a reset controller with the goal of ensuring safe operation, but makes several assumptions that make it difﬁcult to use in the real world: it assumes access to a ground truth reward function, it assumes access to an oracle function that can…”
Learning to Walk via Deep Reinforcement Learning (2018, background): “In the future work, we plan to address these issues by developing a framework for learning policies that are safety aware and can be trained to automatically reset themselves [10, 24].”
Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning (2021, methodology): “Besides, safe policy improvement methods [23, 44, 6, 11] require a separately estimated model to μ to deal with unseen actions.”
Autonomous Reinforcement Learning via Subgoal Curricula (2021, background): “Note, this ˜ r loosely recovers the objectives used in some prior works [19, 6].”
Recovery RL: Safe Reinforcement Learning With Learned Recovery Zones (2020, background): “Han et al. [16] and Eysenbach et al. [10] introduce reset policies which are trained jointly with the task policy to reset the agent to its initial state distribution, ensuring that the task policy only learns behaviors which can be reset [10].”
AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videos (2019, result): “This is similar to prior work on learning reset controllers [20], [21].”
Mo' States Mo' Problems: Emergency Stop Mechanisms from Observation (2019, background): “Some recent approaches have demonstrated empirical success learning when to intervene, either in the form of resetting, collecting expert feedback, or falling back to a safe policy [8, 16, 19, 14].”
Learning to Walk in the Real World with Minimal Human Effort (2020, background): “Many researchers have proposed extensions of the existing deep RL algorithms to address safety, such as learning an additional safety layer that projects raw actions to a safe feasibility set [3, 25, 20, 17], learning the boundary of the safe states with a classifier [38, 49, 23], expanding the iden”
The Regretful Agent: Heuristic-Aided Navigation Through Progress Estimation (2019, background): “On the other hand, relating to the proposed endto-end learned regret module, Leave no Trace [10] learns a forward and a reset policy to reset the environment for preventing the policy entering a non-reversible state.”
Penalizing Side Effects using Stepwise Relative Reachability (2018, methodology): “…2016], penalizing risk [Chow et al., 2015], using intrinsic motivation [Lipton et al., 2016], preserving reversibility [Moldovan and Abbeel, 2012; Eysenbach et al., 2017], etc. Explicitly defined constraints or safe regions tend to be task-specific and require significant human input, so they do…”
Constrained Exploration and Recovery from Experience Shaping (2018, background): “Our approach is thus related to that of (Eysenbach et al. 2018), where a reset policy was learned to return the environment to a safe state for future attempts.”
Measuring and avoiding side effects using relative reachability (2018, methodology): “The two main types of approaches in the literature, ergodicitypreserving safe exploration (Moldovan and Abbeel, 2012; Eysenbach et al., 2017) and low impact (Armstrong and Levinstein, 2017; Taylor et al.”