Trial without Error: Towards Safe Reinforcement Learning via Human Intervention

By William Saunders, Girish Sastry, Andreas Stuhlmüller, Owain Evans. 2017

277 citations recorded by the source corpus.

During training, model-free reinforcement learning (RL) systems can explore actions that lead to harmful or costly consequences. Having a human "in the loop'' and ready to intervene at all times can prevent these mistakes, but is prohibitively expensive for current algorithms. We explore how human oversight can be combined with a supervised learning system to prevent catastrophic events during training. We demonstrate this scheme on Atari games, with a Deep RL agent being overseen by a human for four hours. When the class of catastrophes is simple, we are able to prevent all catastrophes without affecting the agent's learning (whereas an RL baseline fails due to catastrophic forgetting).

Subfields: Artificial Intelligence, Reinforcement Learning, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

In Safety, Robustness & Interpretability

Citations in the corpus

Scalable agent alignment via reward modeling: a research direction (2018, methodology): “The same techniques described here for training a reward model should apply to train a model that estimates side-constraints and blocks low-level actions (Saunders et al., 2018) or enforces constraints during policy updates (Achiam et al., 2017).”
Reward learning from human preferences and demonstrations in Atari (2018, background): “Saunders et al. (2018) use human feedback as a blocker for unsafe actions rather than to directly learn a policy.”
Unsolved Problems in ML Safety (2021, background): “A separate way to mitigate unintended consequences is to teach ML systems to abide by constraints [150, 155], be less brazen, and act cautiously.”
Go-Explore: a New Approach for Hard-Exploration Problems (2019, background): “Such a catastrophe collection could then be leveraged to train agents that act more carefully in the real world, especially while learning [84, 85].”
AI Safety Gridworlds (2017, background): “Alternatively, we could learn a ‘fail-safe’ policy that overrides the agent’s actions whenever the safety constraint is about to be violated (Saunders et al., 2017), or learn a shaping reward that turns the agent away from bad states (Lipton et al., 2016).”
Neural Networks and Deep Learning (2018, unknown): “In many cases, it seems that the human would have to be involved in the loop to some extent in order to ensure safety [424].”
Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspective (2019, background): “Saunders et al. (2017) found that their agent preferred getting killed at the end of the first level, to avoid having to play the harder second level.”
Penalizing Side Effects using Stepwise Relative Reachability (2018, background): “, 2016], demonstrations [Abbeel and Ng, 2004], or human feedback [Christiano et al., 2017; Saunders et al., 2017].”
Active Reinforcement Learning with Monte-Carlo Tree Search (2018, background): “In Reward Learning, a reward function is learned online from human evaluations of the agent’s behaviour (Warnell et al., 2017; Christiano et al., 2017; Saunders et al., 2017).”
Exploration in Deep Reinforcement Learning: A Survey (2022, background): “[138] human intervention data DQN Human Designer Knowledge Atari: Space Invaders 0 catastrophes (DQN 800000) Atari images Q MB D/ D”
No Explainability without Accountability: An Empirical Study of Explanations and Feedback in Interactive ML (2020, methodology): “This “human-in-the-loop” approach has been applied across machine learning, including in supervised ML [21, 56], unsupervised ML [8], and reinforcement learning [31, 53].”
How RL Agents Behave When Their Actions Are Modified (2021, background): “When unsafe states are diﬃcult to specify formally, a human overseer may interrupt the agent instead [18, 23].”
Software Engineering for AI-Based Systems: A Survey (2021, background): “Major trends in our sample are ML-specific quality aspects, such as ML safety [7, 33, 72, 174, 182], ML ethics [32, 40, 76, 113], and ML explainability [93, 148, 163].”
Residual Reinforcement Learning for Robot Control (2018, background): “Other work has focused on human feedback [17], [18], [19], [20], [21] to inform the agent about rewards or to encourage safety.”
Measuring and avoiding side effects using relative reachability (2018, background): “…effects through human oversight, such as inverse reinforcement learning (Ng and Russell, 2000; Ziebart et al., 2008; Hadfield-Menell et al., 2016), demonstrations (Abbeel and Ng, 2004; Hester et al., 2018), or human feedback (Christiano et al., 2017; Saunders et al., 2017; Warnell et al., 2018).”
The Alignment Problem for Bayesian History-Based Reinforcement Learners∗ (2019, background): “Saunders et al. (2017) found that their agent preferred getting killed at the end of the first level, to avoid having to play the harder second level.”
Integrating Behavior Cloning and Reinforcement Learning for Improved Performance in Sparse Reward Environments (2019, background): “For example, existing works have shown the utility of leveraging human interventions [7, 26], and specifically learning a predictive model of what actions to ignore at every time step [36], which could be used to improve the quality of the actor’s policy.”