Avoiding Wireheading with Value Reinforcement Learning

By Tom Everitt, Marcus Hutter. 2016

51 citations recorded by the source corpus.

How can we design good goals for arbitrarily intelligent agents? Reinforcement learning (RL) may seem like a natural approach. Unfortunately, RL does not work well for generally intelligent agents, as RL agents are incentivised to shortcut the reward sensor for maximum reward – the so-called wireheading problem. In this paper we suggest an alternative to RL called value reinforcement learning (VRL). In VRL, agents use the reward signal to learn a utility function. The VRL setup allows us to remove the incentive to wirehead by placing a constraint on the agent’s actions. The constraint is defined in terms of the agent’s belief distributions, and does not require an explicit specification of which actions constitute wireheading.

Subfields: Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Concrete Problems in AI Safety (2016, background): “This general approach is often referred to as distant supervision, and has also received recent attention in the natural language processing community (see e.g. [58, 97] as well as several of the references above).”
The Surprising Creativity of Digital Evolution: A Collection of Anecdotes from the Evolutionary Computation and Artificial Life Research Communities (2018, neutral): “…for perverse outcomes from optimizing reward functions that appear sensible on their surface [4, 11, 37, 134], characterized in that community as problems such as avoiding negative side effects [4, 69], reward hacking (also known as wire-heading) [4, 36], or more generally AI alignment [134].”
Occam's razor is insufficient to infer the preferences of irrational agents (2017, background): “More extreme versions could involve manip-ulative propaganda, drug injections or even coercive brain surgery — a form of human wireheading [Everitt and Hutter, 2016], where the agent changes the human’s behaviour and apparent preferences.”
Alignment for Advanced Machine Learning Systems (2020, background): “Everitt and Hutter (2016) describe a general value learning agent that avoids some potential problems with reinforcement learning and might reproduce approval-directed behavior given a good understanding of how to learn reward functions.”
Self-Modification of Policy and Utility Function in Rational Agents (2016, background): “A companion paper (Everitt and Hutter, 2016) addresses the related problem of agents subverting the evidence they receive, rather than modifying themselves.”
Positive-Unlabeled Reward Learning (2019, background): “However, because the reward supervision often only covers a small part of the state space, RL algorithms may exploit the errors in the reward models to achieve high pseudoreward from unintended behaviors [22].”
Rethinking Formal Models of Partially Observable Multiagent Decision Making (2019, background): “Unobservable rewards also remove incentives to “hide from bad news” or “wirehead” [13].”
The Alignment Problem for Bayesian History-Based Reinforcement Learners∗ (2019, background): “Under some assumptions, a corruption test can be used to remove the data corruption incentive by constraining the agent’s policy search to non-manipulative policies (Armstrong, Ortega, et al., 2018; Everitt and Hutter, 2016).”
RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback (2023, background): “The KL loss helps prevent π RLθ from drifting into a region where it generates language that is highly rewarded by the RM yet consists of low-quality or unnatural language - a phenomenon known as “reward hacking” (Everitt & Hutter, 2016; Amodei et al., 2016).”
RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback (2023, unknown): “The KL loss helps prevent π RLθ from drifting into a region where it generates language that is highly rewarded by the RM yet consists of low-quality or unnatural language - a phenomenon known as “reward hacking” (Everitt and Hutter, 2016; Amodei et al., 2016).”
Understanding Agent Incentives using Causal Influence Diagrams. Part I: Single Action Settings (2019, background): “re, works relating to what we call intervention incentives have been motivated by worries of a powerful reinforcement learning 20 agent tampering with the reward signal (Bostrom, 2014; Everitt, 2018; Everitt and Hutter, 2016; Everitt, Krakovna, et al., 2017), the observation (Ring and Orseau, 2011),”