Self-Modification of Policy and Utility Function in Rational Agents

By Tom Everitt, Daniel Filan, Mayank Daswani, Marcus Hutter. 2016

36 citations recorded by the source corpus.

Any agent that is part of the environment it interacts with and has versatile actuators (such as arms and fingers), will in principle have the ability to self-modify – for example by changing its own source code. As we continue to create more and more intelligent agents, chances increase that they will learn about this ability. The question is: will they want to use it? For example, highly intelligent systems may find ways to change their goals to something more easily achievable, thereby ‘escaping’ the control of their creators. In an important paper, Omohundro (2008) argued that goal preservation is a fundamental drive of any intelligent system, since a goal is more likely to be achieved if future versions of the agent strive towards the same goal. In this paper, we formalise this argument in general reinforcement learning, and explore situations where it fails. Our conclusion is that the self-modification possibility is harmless if and only if the value function of the agent anticipates the consequences of self-modifications and use the current utility function when evaluating the future.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Safety, Robustness & Interpretability

Citations in the corpus

Concrete Problems in AI Safety (2016, background): “Such a suite of environments might serve a benchmarking role similar to that of the bAbI tasks [159], with the eventual goal being to develop a single architecture that can learn to avoid catastrophes in all environments in the suite.”
AI Safety Gridworlds (2017, background): “When self-modifications are performed directly (not via actions in the environment) with predictable consequences on the resulting behavior, it has been shown that the agent can still avoid harming itself (Orseau and Ring, 2011; Hibbard, 2012; Everitt et al., 2016).”
Avoiding Wireheading with Value Reinforcement Learning (2016, background): “Can the CP-VRL results be combined with other AI safety approaches such as self-modiﬁcation (Everitt et al., 2016; Hibbard, 2012), corrigibility (Soares et al., 2015), suicidal agents (Martin et al., 2016), and physicalistic reasoning (Everitt et al., 2015)?”
Understanding Agent Incentives using Causal Influence Diagrams. Part I: Single Action Settings (2019, methodology): “Many more applications of CIDs are provided by Everitt and Hutter (2019) and Everitt et al. (2019).”
Modeling AGI Safety Frameworks with Causal Influence Diagrams (2019, methodology): “An elegant solution to this problem is to use model-based agents that simulate the state sequence likely to result from different policies, and evaluate those state sequences according to the current or initial reward function [Everitt, 2018; Orseau and Ring, 2011; Hibbard, 2012; Schmidhuber, 2007; ”
AGI Safety Literature Review (2018, background): “Everitt, Filan, et al. (2016) formalize this argument, showing that it holds under three non-trivial assumptions: (1) The agent needs to be model-based, and evaluate future scenarios according to its current utility function; (2) the agent needs to be able to predict how self-modiﬁcations aﬀect its…”