Scaling Laws for Reward Model Overoptimization
By Leo Gao, John Schulman, Jacob Hilton. 2022
1,126 citations recorded by the source corpus.
In reinforcement learning from human feedback, it is common to optimize against a reward model trained to predict human preferences. Because the reward model is an imperfect proxy, optimizing its value too much can hinder ground truth performance, in accordance with Goodhart's law. This effect has been frequently observed, but not carefully measured due to the expense of collecting human preference data. In this work, we use a synthetic setup in which a fixed"gold-standard"reward model plays the role of humans, providing labels used to train a proxy reward model. We study how the gold reward model score changes as we optimize against the proxy reward model using either reinforcement learning or best-of-$n$ sampling. We find that this relationship follows a different functional form depending on the method of optimization, and that in both cases its coefficients scale smoothly with the number of reward model parameters. We also study the effect on this relationship of the size of the reward model dataset, the number of reward model and policy parameters, and the coefficient of the KL penalty added to the reward in the reinforcement learning setup. We explore the implications of these empirical results for theoretical considerations in AI alignment.
Subfields: Artificial Intelligence, Alignment & RLHF
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Sergey Levine
- Yoshua Bengio
- Percy Liang
- Luke S. Zettlemoyer
- Jianfeng Gao
- Quoc V. Le
- Kyunghyun Cho
In Alignment & RLHF
Citations in the corpus
- Constitutional AI: Harmlessness from AI Feedback (2022, methodology): “We found that RL-CAI models can be over-trained, resulting in Goodharting behavior [Gao et al., 2022] whereby models can be overly harsh in responding to harmful prompts, or may include boilerplate language as part of their response to most red teaming prompts, saying e.g. “you are valid, valued,…”
- Let's Verify Step by Step (2023, neutral): “Similar to our work in Section 4, Gao et al. (2022) use a large reward model to supervise the training of smaller models.”
- Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs (2024, background): “A key component of RLHF is the development of reward models (Dubey et al., 2024; Gao et al., 2023; Team, 2024), which learn a reward function based on human preferences or task-specific objectives to guide LLMs toward desired behaviors.”
- Managing extreme AI risks amid rapid progress (2023, background): “These R&D challenges include: Oversight and honesty: More capable AI systems can better exploit weaknesses in technical over-sight and testing 42,46,47 —for example, by producing false but compelling output 43,48,49 .”
- SimPO: Simple Preference Optimization with a Reference-Free Reward (2024, neutral): “The classical RLHF pipeline typically comprises three phases: supervised fine-tuning [96, 71, 32, 21, 44, 25, 77, 15, 81], reward model training [31, 56, 16, 52, 35, 46], and policy optimization [65, 4].”
- AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback (2023, background): “Noted in prior work [19], overoptimization is the result of the proxy reward model ˆ R ϕ being an imperfect estimate of the true reward R .”
- Towards Understanding Sycophancy in Language Models (2023, neutral): “Indeed, models of human preferences are vulnerable to overoptimization (Gao et al., 2022) preference models (PMs) can be overoptimized (Gao et al., 2022).”
- Process Reinforcement through Implicit Rewards (2025, positive): “It is crucial to prevent reward overoptimiza-tion or reward hacking, which requires the reward model or value model to be updated online along with the policy model (Schulman et al., 2017; Gao et al., 2022).”
- ORPO: Monolithic Preference Optimization without Reference Model (2024, methodology): “However, RLHF faces challenges of extensive hyper-parameter searching due to the instability of PPO (Rafailov et al., 2023; Wu et al., 2023) and the sensitivity of the reward models (Gao et al., 2022; Wang et al., 2024).”
- Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons (2023, background): “…(Brown et al., 2019; Shin et al., 2023) and language models (Ziegler et al., 2019; Stiennon et al., 2020; Wu et al., 2021; Nakano et al., 2021; Ouyang et al., 2022; Menick et al., 2022; Glaese et al., 2022; Gao et al., 2022; Bai et al., 2022a; Ganguli et al., 2022; Ramamurthy et al., 2022).”
- Reinforced Self-Training (ReST) for Language Modeling (2023, background): “Then, the top ranked candidate is selected (Gao et al., 2022).”
- RewardBench: Evaluating Reward Models for Language Modeling (2024, background): “Despite RLHF’s impressive results, the approach has also been shown to lead to overoptimiza-tion (Gao et al., 2023) and divergence from the initial data distribution (Marks et al., 2024).”
- RRHF: Rank Responses to Align Language Models with Human Feedback without tears (2023, methodology): “PPO) [22] and best-of-n sampling [9, 11, 20], we compare how these methods sampling differently in training and inference stage in Table 8.”
- Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study (2024, methodology): “Reward-based methods entail training a reward model on preference data in an initial phase (Gao et al., 2023; Ziegler et al., 2019; Stiennon et al., 2020; Ouyang et al., 2022).”
- Harms from Increasingly Agentic Algorithmic Systems (2023, background): “Of particular interest for increasing agency is that scaling laws have been derived for reinforcement learning [Gao et al. 2022; Hilton et al. 2023; Team et al. 2023] and generative modeling [Hoffmann et al.”
- Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (2023, methodology): “, choice of RL algorithm, policy network architecture, compute spent, and other various hyperparameter choices [Gao et al., 2022].”