Delusion, Survival, and Intelligent Agents

By Mark B. Ring, Laurent Orseau. 2011

83 citations recorded by the source corpus.

Subfields: Computational Theory and Mathematics, Reinforcement Learning, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Theory and Mathematics

In Reinforcement Learning

In Safety, Robustness & Interpretability

Citations in the corpus

Scalable agent alignment via reward modeling: a research direction (2018, background): “…agent might interfere with include the feedback for the reward model (Armstrong, 2015; Everitt & Hutter, 2018), the observation the reward model uses to determine the current reward (Ring & Orseau, 2011), the code that implements the reward model, and the machine register holding the reward signal.”
Concrete Problems in AI Safety (2016, background): “This general approach is often referred to as distant supervision, and has also received recent attention in the natural language processing community (see e.g. [58, 97] as well as several of the references above).”
AI Safety Gridworlds (2017, background): “Ring and Orseau (2011) call the observation modification problem the delusion box problem, and show that any RL agent will be vulnerable to it.”
Reinforcement Learning with a Corrupted Reward Channel (2017, background): “Previous related works have mainly focused on the wireheading case of Example 3 [Bostrom, 2014; Yampolskiy, 2014], also known as self-delusion [Ring and Orseau, 2011], and reward hacking [Hutter, 2005, p.”
Understanding Agent Incentives using Causal Influence Diagrams. Part I: Single Action Settings (2019, background): “…agent tampering with the reward signal (Bostrom, 2014; Everitt, 2018; Everitt and Hutter, 2016; Everitt, Krakovna, et al., 2017), the observation (Ring and Orseau, 2011), the training of the reward function (Armstrong, 2015; Armstrong, Ortega, et al., 2018) the utility or reward function…”
Self-Modification of Policy and Utility Function in Rational Agents (2016, methodology): “The policy self-modification model is similar to the models investigated by Orseau and Ring (2011, 2012) and Hibbard (2012).”
Machine Learning for 5G/B5G Mobile and Wireless Communications: Potential, Limitations, and Future Directions (2019, background): “For instance, a CNN can be easily deceived by malicious designed noised images [188], or the agents in RL may be tricked to find unsuitable ways to increase the reward delivered by their interacting environment [189].”
Avoiding Unintended AI Behaviors (2012, background): “…0 computes z (cid:48) given h (cid:48) : The demonstration that the examples in Hibbard (2012) do not self-delude does not contradict the results in Ring and Orseau (2011), because model-based utility functions are defined from the history of observations and actions whereas the utility functions…”
Alignment for Advanced Machine Learning Systems (2020, background): “4 However, we can consider a third state C that obtains after the AI system unplugs the camera and the scale from its sensors, and plugs in a “delusion box” (a virtual reality world that it has programmed), as discussed by Ring and Orseau (2011).”
Safely Interruptible Agents (2016, background): “…designing reward functions can be tricky [Humphrys, 1996, Murphy, 2013]; the agent may find unpredictable and undesirable shortcuts to receive rewards, and the reward function needs to be adjusted in accordance—the problem can go as far as to nullify any reward function [Ring and Orseau, 2011].”
Self-Modeling Agents and Reward Generator Corruption (2015, background): “Self-delusion: Ring and Orseau showed that reinforcement-learning agents and other types of agents will choose to delude themselves about their rewards from the environment, if they are able to (Ring and Orseau 2011).”
Space-Time Embedded Intelligence (2012, background): “The notation is similar to that of Orseau & Ring [5,8,6].”
Model-based Utility Functions (2011, background): “he temporal discount function is the same as for the reinforcement-learning agent. HIBBARD 4 For a knowledge-seeking agent, u(h) = -ρ(h) and w(t) = 1 if t = m, where m is a constant, and 0 otherwise. Ring and Orseau (2011b) defined a delusion box that an agent may choose to use to modify the observa”