Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspective

By Tom Everitt, Marcus Hutter. 2019

156 citations recorded by the source corpus.

Can an arbitrarily intelligent reinforcement learning agent be kept under control by a human user? Or do agents with sufficient intelligence inevitably find ways to shortcut their reward signal? This question impacts how far reinforcement learning can be scaled, and whether alternative paradigms must be developed in order to build safe artificial general intelligence. In this paper, we use an intuitive yet precise graphical model called causal influence diagrams to formalize reward tampering problems. We also describe a number of modifications to the reinforcement learning objective that prevent incentives for reward tampering. We verify the solutions using recently developed graphical criteria for inferring agent incentives from causal influence diagrams. Along the way, we also compare corrigibility and self-preservation properties of the various solutions, and discuss how they can be combined into a single agent without reward tampering incentives.

Subfields: Safety Research, Alignment & RLHF, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Safety Research

In Alignment & RLHF

In Reinforcement Learning

Citations in the corpus

Alignment of Language Agents (2021, methodology): “Secondly, even if the supplied objective is correct, a capable agent may still exhibit undesired behaviour due to secondary objectives that arise in pursuit of its primary objective, such as tampering with its feedback channel (Everitt et al., 2021b).”
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (2023, background): “These incentives to mislead also connect to broader worries about manipulation [Kenton et al., 2021, Carroll et al., 2023, Everitt et al., 2021].”
ReFT: Reasoning with Reinforced Fine-Tuning (2024, unknown): “The underlying reward hacking phenomenon severely tampers the model training (Everitt et al., 2021).”
Agent Incentives: A Causal Perspective (2021, methodology): “…diagrams have already aided in the understanding of incentive problems such as an agent’s redirectability, ambition, tendency to tamper with reward, and other properties (Armstrong et al. 2020; Holtman 2020; Cohen, Vellambi, and Hutter 2020; Everitt et al. 2019a,b; Langlois and Everitt 2021).”
Reasoning Models Don't Always Say What They Think (2025, unknown): “However, a substantial open challenge in RL is reward hacking, where models learn spurious correlations that yield high rewards during training but do not generalize to test examples (Everitt et al., 2021; Krakovna et al., 2020; Pan et al., 2022; Denison et al., 2024).”
Kimi k1.5: Scaling Reinforcement Learning with LLMs (2025, unknown): “To avoid potential reward hacking (Everitt et al. 2021; Pan et al. 2022), we need to ensure that both the reasoning process and the final answer of each prompt can be accurately verified.”
The alignment problem from a deep learning perspective (2022, background): “However, in principle feedback mechanism ﬁxation could occur without any reward misspeciﬁcation, since strategies for directly inﬂuencing feedback mechanisms (like reward tampering [Everitt et al., 2021]) can receive high reward for any reward function.”
Defining and Characterizing Reward Hacking (2022, unknown): “Reward tampering (Everitt et al., 2017; Kumar et al., 2020; Uesato et al., 2020; Everitt et al., 2021) can be viewed as a special case of reward hacking, and refers to an agent corrupting the process generating reward signals, e.g. by tampering with sensors, memory registers storing the reward…”