Model-based Utility Functions

By Bill Hibbard. 2011

51 citations recorded by the source corpus.

Abstract Orseau and Ring, as well as Dewey, have recently described problems, including self-delusion, with the behavior of agents using various definitions of utility functions. An agent's utility function is defined in terms of the agent's history of interactions with its environment. This paper argues, via two examples, that the behavior problems can be avoided by formulating the utility function in two steps: 1) inferring a model of the environment from interactions, and 2) computing utility as a function of the environment model. Basing a utility function on a model that the agent must learn implies that the utility function must initially be expressed in terms of specifications to be matched to structures in the learned model. These specifications constitute prior assumptions about the environment so this approach will not work with arbitrary environments. But the approach should work for agents designed by humans to act in the physical world. The paper also addresses the issue of self-modifying agents and shows that if provided with the possibility to modify their utility functions agents will not choose to do so, under some usual assumptions.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Reinforcement Learning

Citations in the corpus

Concrete Problems in AI Safety (2016, background): “Robot locomotion or industrial control tasks might be a natural candidate for such experiments.”
AI Safety Gridworlds (2017, background): “When self-modifications are performed directly (not via actions in the environment) with predictable consequences on the resulting behavior, it has been shown that the agent can still avoid harming itself (Orseau and Ring, 2011; Hibbard, 2012; Everitt et al., 2016).”
Modeling AGI Safety Frameworks with Causal Influence Diagrams (2019, methodology): “An elegant solution to this problem is to use model-based agents that simulate the state sequence likely to result from different policies, and evaluate those state sequences according to the current or initial reward function [Everitt, 2018; Orseau and Ring, 2011; Hibbard, 2012; Schmidhuber, 2007; ”
AGI Safety Literature Review (2018, background): “Hibbard (2012) and Orseau and Ring (2011) also study the utility self-preservation argument.”
Alignment for Advanced Machine Learning Systems (2020, background): “Dewey (2011) and Hibbard (2012) have attempted to extend the AIXI framework of Hutter (2005) so that it learns a utility function over world-states instead of interpreting a certain portion of its percepts as a reward primitive.”
The Alignment Problem for Bayesian History-Based Reinforcement Learners∗ (2019, methodology): “V CUt, adds no incentive for avoiding selfcorruption; V CAt, does add an incentive for avoiding self-corruption (Everitt, Filan, et al., 2016; Hibbard, 2012).”
Self-Modification of Policy and Utility Function in Rational Agents (2016, background): “The main advantage of utility functions over RL is that the agent’s actions can be incorporated into the goal specification, which can prevent self-delusion problems such as the agent manipulating the reward signal (Everitt and Hutter, 2016; Hibbard, 2012; Ring and Orseau, 2011).”
Self-Modeling Agents and Reward Generator Corruption (2015, methodology): “We define a utility function uhuman_values(hm, hx, h), which is the utility of history h, from the perspective of humans at history hx, as modeled by λ(hm) (Hibbard 2012a; 2012b; 2014b).”
Understanding Agent Incentives using Causal Influence Diagrams. Part I: Single Action Settings (2019, background): “ Orseau, 2011), the training of the reward function (Armstrong, 2015; Armstrong et al., 2018; Everitt and Hutter, 2019) the utility or reward function (Everitt et al., 2016; Everitt and Hutter, 2019; Hibbard, 2012; Omohundro, 2008; Orseau and Ring, 2011; Schmidhuber, 2007), or a shut-down signal (Ha”
Utility function security in artificially intelligent agents (2014, background): “Superintelligent AIs of tomorrow will know the difference between the goal and its proxy measure and are believed to be safe by many experts (Dewey, 2011; Hibbard, 2011; Omohundro, 2008; Tyler, 2011c; Yudkowsky, 2011) because they will choose not to wirehead as that does not get them any closer to t”
Avoiding Wireheading with Value Reinforcement Learning (2016, background): “This section considers the argument made by Hibbard (2012) that the wire-heading problem is avoided by utility agents.”