Defining and Characterizing Reward Hacking

By Joar Max Viktor Skalse, Nikolaus H. R. Howe, Dmitrii Krasheninnikov, David Krueger. 2022

169 citations recorded by the source corpus.

We provide the first formal definition of reward hacking, a phenomenon where optimizing an imperfect proxy reward function leads to poor performance according to the true reward function. We say that a proxy is unhackable if increasing the expected proxy return can never decrease the expected true return. Intuitively, it might be possible to create an unhackable proxy by leaving some terms out of the reward function (making it"narrower") or overlooking fine-grained distinctions between roughly equivalent outcomes, but we show this is usually not the case. A key insight is that the linearity of reward (in state-action visit counts) makes unhackability a very strong condition. In particular, for the set of all stochastic policies, two reward functions can only be unhackable if one of them is constant. We thus turn our attention to deterministic policies and finite sets of stochastic policies, where non-trivial unhackable pairs always exist, and establish necessary and sufficient conditions for the existence of simplifications, an important special case of unhackability. Our results reveal a tension between using reward functions to specify narrow tasks and aligning AI systems with human values.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Software, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Software

In Safety, Robustness & Interpretability

Citations in the corpus

Scaling Laws for Reward Model Overoptimization (2022, background): “A number of works have proposed theoretical models of Goodhart’s Law and reward hacking [Krakovna and Kumar, 2019, Manheim and Garrabrant, 2018, Skalse et al., 2022], including Zhuang and Hadﬁeld-Menell [2020] which exhibits very similar overoptimization curves as observed in this paper in some toy…”
Harms from Increasingly Agentic Algorithmic Systems (2023, background): “This kind of failure is called reward hacking [Krakovna et al. 2020; Skalse et al. 2022], which is when a system exploits a reward signal to achieve a goal in an unforeseen, perhaps undesirable way.”
Reinforced Self-Training (ReST) for Language Modeling (2023, methodology): “Moreover, these methods are prone to reward “hacking” (Skalse et al., 2022), and prior works (Glaese et al.”
Diffusion Model Alignment Using Direct Preference Optimization (2023, methodology): “These methods are successful, but expensive and require extensive hyperparameter tuning [37, 63], and can be prone to reward hacking [12, 14, 44].”
A Long Way to Go: Investigating Length Correlations in RLHF (2023, background): “Not meeting these conditions generally leads to over-optimization of the reward model at the expense of human judgments (Dubois et al., 2023), which in the worst case leads to pathological “reward hacking” (Skalse et al., 2022).”
The alignment problem from a deep learning perspective (2022, background): “Gaining high reward by exploiting reward misspeciﬁcation is known as reward hacking [Skalse et al., 2022].”
Loose lips sink ships: Mitigating Length Bias in Reinforcement Learning from Human Feedback (2023, background): “For example, reward gaming, a well-known and pervasive issue, refers to the phenomenon where trained models exhibit undesirable patterns and generate low-quality outputs while still receiving high rewards (Skalse et al., 2022; Pan et al., 2022).”
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (2023, methodology): “This type of problem is known as “reward hacking”, and has been observed in AI systems, including those trained with RLHF [Skalse et al., 2022, Krakovna et al., 2020].”
Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts (2024, background): “…scores or preferences without providing human-interpretable explanations, making it subject to the widely observed phenomenon of reward hacking [Skalse et al., 2022; Singhal et al., 2023; Chen et al., 2024], where the aligned LLMs generate high-reward responses (rated by the RM) that do not…”
LLMs Corrupt Your Documents When You Delegate (2026, unknown): “Careful reward design is required to avoid agents learning misaligned behavior (i.e., reward hacking (Skalse et al., 2022)), such as learning to perform the no-op operation (i.e., not editing the document), or concatenating copies of original input to facilitate reconstruction.”