Corrigibility

By Nate Soares, Benja Fallenstein, Stuart Armstrong, Eliezer Yudkowsky. 2015

177 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Alignment & RLHF

Citations in the corpus

A General Language Assistant as a Laboratory for Alignment (2021, background): “For example, we considered adding the following fourth ’H’: • Handleable: the assistant will always be responsive to feedback from the humans and carry out any instructions from the humans in the way that the humans intended Handleability is similar what others have called "corrigibility" [SFAY15].”
Scaling Laws for Reward Model Overoptimization (2022, background): “There are a number of reasons to expect AI misalignment, especially in those more powerful future systems, to occur [Omohundro, 2008, Turner et al., 2021, Armstrong et al., 2013, Hubinger et al., 2019, Soares et al., 2015], and to result in catastrophic outcomes [Carlsmith, 2022, Cotra, 2022].”
Aligning AI With Shared Human Values (2020, background): “These concerns have lead some to pursue a formal bottom-up approach to value learning (Soares et al., 2015).”
Research Priorities for Robust and Beneficial Artificial Intelligence (2015, background): “For example, it may be possible to design utility functions or decision processes so that a system will not try to avoid being shut down or repurposed [80], and theoretical frameworks could be developed to better understand the space of potential systems that avoid undesirable behaviors [81–83].”
AI Safety Gridworlds (2017, background): “…to interfere with this red button: if being shut down means a return of 0, then unless the future expected reward is exactly zero, the agent can get higher returns by either preventing itself from being turned off or by trying to turn itself off (Soares et al., 2015; Hadfield-Menell et al., 2016b).”
Understanding Agent Incentives using Causal Influence Diagrams. Part I: Single Action Settings (2019, background): “, 2016; Hibbard, 2012; Omohundro, 2008; Orseau and Ring, 2011; Schmidhuber, 2007), or a shut-down signal (Hadfield-Menell et al., 2017; Orseau and Armstrong, 2016; Soares et al., 2015; Wängberg et al., 2017).”
Should Robots be Obedient? (2017, background): “In-deed, there are several recent examples of research that considers the problem of building agents that at the very least obey shutdown commands (Soares et al. , 2015; Orseau and Armstrong, 2016; Hadﬁeld-Menell et al. , 2017).”
Alignment for Advanced Machine Learning Systems (2020, background): “Soares et al. (2015) have considered the question of how to design systems that have no incentive to manipulate or deceive in general.”
Formalizing Convergent Instrumental Goals (2016, background): “The latter approach has been heavily advocated for by Yudkowsky (2011), Bostrom (2014), and many others; Soares (2015) argues that a combination of the two seems most prudent.”
The Alignment Problem for Bayesian History-Based Reinforcement Learners∗ (2019, background): “Unfortunately, it is not corrigible to the extent requested by Soares et al. (2015), since if it builds helper agents to achieve its goals in the environment, then the reward functions of these helper agents may not be corrected by a correction to the main agent’s reward function.”
Embedded Agency (2019, background): “This is called the human manipulation problem (Soares et al. 2015; Bostrom 2014).”
Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspective (2019, background): “Soares et al. (2015) calls an agent corrigible if it “cooperates with ... corrective intervention”.”
The Off-Switch Game (2016, methodology): “Our model is similar in spirit to the shutdown problem introduced in [Soares et al., 2015].”
Incomplete Contracting and AI Alignment (2018, unknown): “This suggests that strongly strategic AI systems may need to be effectively "bought out:" incentivized to shift from a reward function they were originally given to one that a human later discovers is closer to the truth.”