The Off-Switch Game

By Dylan Hadfield-Menell, Anca Dragan, Pieter Abbeel, Stuart J. Russell. 2016

206 citations recorded by the source corpus.

It is clear that one of the primary tools we can use to mitigate the potential risk from a misbehaving AI system is the ability to turn the system off. As the capabilities of AI systems improve, it is important to ensure that such systems do not adopt subgoals that prevent a human from switching them off. This is a challenge because many formulations of rational agents create strong incentives for self-preservation. This is not caused by a built-in instinct, but because a rational agent will maximize expected utility and cannot achieve whatever objective it has been given if it is dead. Our goal is to study the incentives an agent has to allow itself to be switched off. We analyze a simple game between a human H and a robot R, where H can press R's off switch but R can disable the off switch. A traditional agent takes its reward function for granted: we show that such agents have an incentive to disable the off switch, except in the special case where H is perfectly rational. Our key insight is that for R to want to preserve its off switch, it needs to be uncertain about the utility associated with the outcome, and to treat H's actions as important observations about that utility. (R also has no incentive to switch itself off in this setting.) We conclude that giving machines an appropriate level of uncertainty about their objectives leads to safer designs, and we argue that this setting is a useful generalization of the classical AI paradigm of rational agents.

Subfields: Computational Theory and Mathematics, Management Science and Operations Research, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Theory and Mathematics

In Management Science and Operations Research

In Safety, Robustness & Interpretability

Citations in the corpus

Alignment of Language Agents (2021, background): “Another class of approaches (Hadfield-Menell et al., 2016, 2017b) contains an agent which is uncertain about its objective, and aims for the agent to correctly calibrate its beliefs about it, and in doing so avoid gaming it.”
Scalable agent alignment via reward modeling: a research direction (2018, background): “…Asri et al., 2016), actions (Grifﬁth et al., 2013), value (Knox & Stone, 2009), advantage (MacGlashan et al., 2017), or preferences over trajectories (Fürnkranz et al., 2012; Akrour et al., 2012, 2014; Wirth et al., 2017); and providing an explicit objective function (Hadﬁeld-Menell et al., 2017b).”
Unsolved Problems in ML Safety (2021, background): “Likewise, researchers could create agents that properly account for their lack of knowledge of the true objective [69] and avoid disrupting parts of the environment whose value is unclear [190, 102, 157].”
Inverse Reward Design (2017, background): “…uses human demonstrations (Ng & Russell, 2000; Ziebart et al., 2008), shared autonomy uses human operator control signals (Javdani et al., 2015), preference-based reward learning uses answers to comparison queries (Jain et al., 2015), and even what the human wants (Hadﬁeld-Menell et al., 2017).”
AI Safety Gridworlds (2017, background): “However, looking beyond pure RL agents, Everitt et al. (2017) argue that many RL-inspired frameworks such as cooperative inverse RL (Hadfield-Menell et al., 2016a), learning from human preferences (see e.g. Akrour et al., 2012, Wilson et al., 2012 and Christiano et al., 2017), and learning values…”
Reinforcement Learning with a Corrupted Reward Channel (2017, methodology): “The more recently proposed (cooperative) IRL framework may be seen as an approach to build CR agents [Hadfield-Menell et al., 2016, 2017].”
Consequences of Misaligned AI (2021, background): “We need to ensure, for example, that the robot does not hinder the human’s ability to adjust the proxy utility function (16).”
Managing extreme AI risks amid rapid progress (2023, methodology): “To avoid human intervention 35 , they might copy their algorithms across global server networks 36 , as computer worms do.”
Should Robots be Obedient? (2017, background): “In-deed, there are several recent examples of research that considers the problem of building agents that at the very least obey shutdown commands (Soares et al. , 2015; Orseau and Armstrong, 2016; Hadﬁeld-Menell et al. , 2017).”
Symbols as a Lingua Franca for Bridging Human-AI Chasm for Explainable and Advisable AI Systems (2021, unknown): “One way to avoid such biases may be to ensure that the system always maintains some uncertainty in any hypotheses it learns from the human input (Hadfield-Menell et al. 2017).”
Incomplete Contracting and AI Alignment (2018, background): “One interpretation of [18] is as a model not only of the incentive of a robot to disable its off switch but also of its incentive to share with a human the information needed to exercise off-switch authority.”
An Overview of Catastrophic AI Risks (2023, background): “An example by computer scientist Stuart Russell illustrates the potential for instrumental goals to emerge in a wide range of AI systems [116].”
Understanding Agent Incentives using Causal Influence Diagrams. Part I: Single Action Settings (2019, background): “, 2016; Hibbard, 2012; Omohundro, 2008; Orseau and Ring, 2011; Schmidhuber, 2007), or a shut-down signal (Hadfield-Menell et al., 2017; Orseau and Armstrong, 2016; Soares et al., 2015; Wängberg et al., 2017).”
Feature Expansive Reward Learning: Rethinking Human Input (2020, methodology): “work must investigate ways in which the robot can determine whether to accept or reject the newly learned feature. Although this is still being studied, we are hopeful that advances in AI safety like [27] could help ensure that these adverse scenarios don’t happen. Acknowledgments and Disclosure of ”