Safely Interruptible Agents

By Laurent Orseau, Stuart Armstrong. 2016

150 citations recorded by the source corpus.

Subfields: Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Safety, Robustness & Interpretability

Citations in the corpus

Scalable agent alignment via reward modeling: a research direction (2018, background): “…for speciﬁcation problems include the following undesirable incentives (see also Omohundro, 2008): • Off-switch problems (Soares et al., 2015; Orseau & Armstrong, 2016; Hadﬁeld-Menell et al., 2017a): the agent is typically either incentivized to turn itself off or to prevent itself from…”
Concrete Problems in AI Safety (2016, background): “For example, a designer might notice that under ordinary circumstances, a cleaning robot’s success in cleaning up the office is proportional to the rate at which it consumes cleaning supplies, such as bleach.”
Safe Policy Improvement with Baseline Bootstrapping (2017, background): “Other related works High-Confidence PI refers to the family of algorithms introduced in Paduraru (2013); Mandel et al.”
The ethics of algorithms: Mapping the debate (2016, neutral): “Algorithms, particularly those embedded in robotics, can for instance be made safely interruptible insofar as harmful actions can be discouraged without the algorithm being encouraged to deceive human users to avoid further interruptions (Orseau and Armstrong, 2016).”
AI Safety Gridworlds (2017, methodology): “This general instance of the red button problem is called safe interruptibility (Orseau and Armstrong, 2016): How can we design agents that neither seek nor avoid interruptions?”
Penalizing Side Effects using Stepwise Relative Reachability (2018, methodology): “We also examine the effects of impact measure design choices on whether the agent is interruptible [Orseau and Armstrong, 2016], using the Survival Incentive environment introduced in Turner et al. [2019]. In this environment, the agent has the option to disable an off switch, which prevents the epi”
How RL Agents Behave When Their Actions Are Modified (2021, methodology): “Combined with the convergence result of Theorem 6 in Section 4, this generalizes the Q-learning result of Orseau and Armstrong [18] to arbitrary policy-dependent action modiﬁcations: it is possible to train agents that ignore applied action modiﬁcations and one approach is as simple as using algorit”
AGI Safety Literature Review (2018, methodology): “For example, this technique can be used to construct variants of popular RL algorithms that do not learn to prevent interruptions (Orseau and Armstrong, 2016).”
Should Robots be Obedient? (2017, background): “Indeed, there are several recent examples of research that considers the problem of building agents that at the very least obey shutdown commands [Soares et al., 2015; Orseau and Armstrong, 2016; Hadfield-Menell et al., 2017].”
Society-in-the-loop: programming the algorithmic social contract (2017, background): “Some work is underway to ensure AI cannot learn to disable their own kill-switch [56].”
Dead-ends and Secure Exploration in Reinforcement Learning (2019, background): “…& Fernández, 2015) is an overloaded term, as it may be considered with respect to parametric uncertainty (Thomas, 2015; Petrik et al., 2016; Laroche et al., 2019), internal uncertainty (Altman, 1999; Carrara et al., 2019), or interruptibility (Orseau & Armstrong, 2016; Guerraoui et al., 2017).”
Unity: A General Platform for Intelligent Agents (2018, background): “An alternate line of work investigates how to design agents that don’t learn to avoid being interrupted by humans given that it may prevent them from receiving future reward (Orseau and Armstrong, 2016).”
Dynamic Safe Interruptibility for Decentralized Multi-Agent Reinforcement Learning (2017, background): “(Interruptibility [16]) Given an interruption scheme < I, θ, π >, the interruption operator at time t is defined by INT (π) = π with probability I ·θt and π otherwise.”
Towards Safe Artificial General Intelligence (2018, methodology): “Reward function misspecification. Misalignment can also be caused by a misspecification of the initial reward function R̃0. Examples: (c) (Real) CoastRunners is a video game where the desired behavior is winning a boat race. Clark and Amodei (2016) describes how an agent trained on CoastRunners foun”
Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspective (2019, background): “not accidentally learn a TI-aware model of the world. For example, on-policy algorithms such as SARSA may be naturally TI-aware, while oﬀ-policy agents such as Q-learning may be naturally TI-unaware (Orseau and Armstrong, 2016). 14Belief here refers to the model that the agent’s action is optimized ”
The Off-Switch Game (2016, background): “[Orseau and Armstrong, 2016] consider the impact of human interference on the learning process.”
Incomplete Contracting and AI Alignment (2018, unknown): “AI safety problems such as safe interruptibility [37], the off-switch game [18] and corrigibility [47] are also alignment problems: these are efforts to ensure that AI agents value shut down or modification of their reward functions in the same way that humans do or at least are indifferent to such…”
Understanding Agent Incentives using Causal Influence Diagrams. Part I: Single Action Settings (2019, background): “lity or reward function (Everitt et al., 2016; Everitt and Hutter, 2019; Hibbard, 2012; Omohundro, 2008; Orseau and Ring, 2011; Schmidhuber, 2007), or a shut-down signal (Hadﬁeld-Menell et al., 2017; Orseau and Armstrong, 2016; Soares et al., 2015; W¨angberg et al., 2017). Another example is that of”