Utility function security in artificially intelligent agents

By Roman V Yampolskiy. 2014

51 citations recorded by the source corpus.

The notion of ‘wireheading’, or direct reward centre stimulation of the brain, is a well-known concept in neuroscience. In this paper, we examine the corresponding issue of reward (utility) function integrity in artificially intelligent machines. We survey the relevant literature and propose a number of potential solutions to ensure the integrity of our artificial assistants. Overall, we conclude that wireheading in rational self-improving optimisers above a certain capacity remains an unsolved problem despite opinion of many that such machines will choose not to wirehead. A relevant issue of literalness in goal setting also remains largely unsolved and we suggest that the development of a non-ambiguous knowledge transfer language might be a step in the right direction.

Subfields: Cognitive Neuroscience, Electrical and Electronic Engineering, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

1. John Schulman
2. Jacob Steinhardt

In Electrical and Electronic Engineering

1. Dario Amodei
2. Dandelion Mané
3. John Schulman
4. Jacob Steinhardt

In Safety, Robustness & Interpretability

1. John Schulman
2. Dario Amodei
3. Dandelion Mané
4. Jacob Steinhardt
5. Yarin Gal
6. Richard Ngo
7. Shane Legg
8. Dan Hendrycks

Citations in the corpus

• Concrete Problems in AI Safety (2016, background): “This particular failure mode is often called “wireheading” [49, 127, 42, 67, 165].”