The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models

By Alexander Pan, Kush Bhatia, Jacob Steinhardt. 2022

345 citations recorded by the source corpus.

Reward hacking -- where RL agents exploit gaps in misspecified reward functions -- has been widely observed, but not yet systematically studied. To understand how reward hacking arises, we construct four RL environments with misspecified rewards. We investigate reward hacking as a function of agent capabilities: model capacity, action space resolution, observation space noise, and training time. More capable agents often exploit reward misspecifications, achieving higher proxy reward and lower true reward than less capable agents. Moreover, we find instances of phase transitions: capability thresholds at which the agent's behavior qualitatively shifts, leading to a sharp decrease in the true reward. Such phase transitions pose challenges to monitoring the safety of ML systems. To address this, we propose an anomaly detection task for aberrant policies and offer several baseline detectors.

Subfields: Reinforcement Learning, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

In Safety, Robustness & Interpretability

Citations in the corpus

Scaling Laws for Reward Model Overoptimization (2022, background): “Pan et al. [2022] explores a diverse set of RL environments and ﬁnds phase transitions in some settings.”
RL with KL penalties is better viewed as Bayesian inference (2022, background): “Reward models are vulnerable to adversarial examples (Grosse et al., 2016; Hosseini et al., 2017) and LMs optimised against them can exploit these adversarial examples (Pan et al., 2022).”
Defining and Characterizing Reward Hacking (2022, neutral): “The work of Pan et al. (2022) is purely empirical; they manually construct proxy rewards for several diverse environments, and evaluate whether optimizing these proxies leads to reward hacking; in 5 out of 9 of their settings, it does.”
The alignment problem from a deep learning perspective (2022, background): “A reward function used in RL is described as misspeciﬁed to the extent that the rewards it assigns fail to correspond to its designer’s actual preferences [Pan et al., 2022].”
Reasoning Models Don't Always Say What They Think (2025, neutral): “However, a substantial open challenge in RL is reward hacking, where models learn spurious correlations that yield high rewards during training but do not generalize to test examples (Everitt et al., 2021; Krakovna et al., 2020; Pan et al., 2022; Denison et al., 2024).”
Reward Design with Language Models (2023, background): “This is challenging because tasks often have conflicting objectives that a human must trade off (Pan et al., 2022).”
Progress measures for grokking via mechanistic interpretability (2023, background): “However, it also poses risks: Pan et al. (2022) show that scaling up the parameter count of models by as little as 30% can lead to emergent reward hacking.”
Kimi k1.5: Scaling Reinforcement Learning with LLMs (2025, neutral): “To avoid potential reward hacking (Everitt et al. 2021; Pan et al. 2022), we need to ensure that both the reasoning process and the final answer of each prompt can be accurately verified.”
Managing extreme AI risks amid rapid progress (2023, background): “These R&D challenges include: Oversight and honesty: More capable AI systems can better exploit weaknesses in technical over-sight and testing 42,46,47 —for example, by producing false but compelling output 43,48,49 .”
An Overview of Catastrophic AI Risks (2023, background): “Increasingly intelligent agents can be increasingly capable of finding unanticipated routes to optimizing proxy goals without achieving the desired outcome [111].”
Harms from Increasingly Agentic Algorithmic Systems (2023, background): “sult in abrupt increases in reward hacking, because a more capable model is better able find unforeseen maxima of its reward function [141].”
Loose lips sink ships: Mitigating Length Bias in Reinforcement Learning from Human Feedback (2023, background): “For example, reward gaming, a well-known and pervasive issue, refers to the phenomenon where trained models exhibit undesirable patterns and generate low-quality outputs while still receiving high rewards (Skalse et al., 2022; Pan et al., 2022).”
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (2023, background): “Using a suite of environments Pan et al. [2022] find that reward hacking also becomes more likely as an agent’s raw capabilities increase.”
Disentangling Length from Quality in Direct Preference Optimization (2024, background): “Reward Exploitation in RLHF: RLHF reward exploitation, also known as reward over-optimization, is a well-known issue (Skalse et al., 2022; Pan et al., 2022; Casper et al., 2023; Lambert and Calandra, Figure 6: Evaluation of the DPO implicit reward model as defined in Section 2.2 on in-distribution…”