A Long Way to Go: Investigating Length Correlations in RLHF
By Prasann Singhal, Tanya Goyal, Jiacheng Xu, Greg Durrett. 2023
284 citations recorded by the source corpus.
Great success has been reported using Reinforcement Learning from Human Feedback (RLHF) to align large language models, with open preference datasets enabling wider experimentation, particularly for"helpfulness"in tasks like dialogue and web question answering. Alongside these improvements, however, RLHF also often drives models to produce longer outputs. This paper demonstrates, on three diverse settings, that optimizing for response length is, much more than previously thought, a significant factor behind RLHF. Studying the strategies RL optimization uses to maximize reward, we find improvements in reward to largely be driven by increasing response length, instead of other features. Indeed, we find that even a purely length-based reward reproduces most downstream RLHF improvements over supervised fine-tuned models. Testing a comprehensive set of length-countering interventions, we identify the dominant source of these biases to be reward models, which, by studying training dynamics, we find are non-robust and easily influenced by length biases in preference data.
Subfields: Artificial Intelligence, Information Systems, Alignment & RLHF
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Percy Liang
- Mohit Bansal
- Luke S. Zettlemoyer
- Jianfeng Gao
- Yejin Choi
- Quoc V. Le
- Danqi Chen
- Yoshua Bengio
In Information Systems
- Percy Liang
- Danqi Chen
- Yu Meng
- Tianyi Zhou
- Tom Goldstein
- Tatsunori B. Hashimoto
- Bryan Catanzaro
- Stefano Ermon
In Alignment & RLHF
- Rafael Rafailov
- Yu Meng
- Percy Liang
- Tatsunori B. Hashimoto
- Wei Xiong
- Chelsea Finn
- Ryan Park
- Stefano Ermon
Citations in the corpus
- Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators (2024, methodology): “We see that proprietary models, which often generate shorter responses, perform much better on AlpacaEval-LC, and the biggest rank losses are in open-source models that have gone through the RLHF process Ouyang et al. (2022).”
- TÜLU 3: Pushing Frontiers in Open Language Model Post-Training (2024, unknown): “…5.4) that length-normalized DPO works best, which uses the following objective: As seen, this is simply the DPO objective (Eq 5), but with log-probabilities normalized for length, which intuitively aids with mitigating the length bias common in human and model preferences [Singhal et al., 2024].”
- Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study (2024, background): “It is well-known that PPO could exploit potential failures in the learned reward model to achieve high rewards without meeting the actual human preference, often manifested as erroneous (Lewis et al., 2017) or overly complex outputs (Singhal et al., 2023).”
- Disentangling Length from Quality in Direct Preference Optimization (2024, background): “(Ramamurthy et al., 2023) suggest setting a simple discount factor, which improves naturalness of the generated language, (Singhal et al., 2023) carry out an extensive study of length correlations in classical RLHF and suggest a number of mitigating approaches.”
- SimPO: Simple Preference Optimization with a Reference-Free Reward (2024, unknown): “Further research has also demonstrated that RLHF can lead to biased outcomes, such as verbose outputs from the model [28, 66, 80].”
- ODIN: Disentangled Reward Mitigates Hacking in RLHF (2024, methodology): “Singhal et al. [2023] adds a length penalty proportional to the response length using the standard deviation of reward as the coefficient.”
- RewardBench: Evaluating Reward Models for Language Modeling (2024, background): “Other works, on the other hand, analyze the reward model, such as Singhal et al. (2023), who look at the training dynamics of RMs.”
- Evaluating Large Language Models at Evaluating Instruction Following (2023, background): “Table 3 shows that these three models fall significantly short on LLMB AR , even on N ATURAL , suggesting that current reward models and preference models struggle to identify instruction-following outputs, a finding in line with Shen et al. (2023); Singhal et al. (2023).”
- Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts (2024, background): “…without providing human-interpretable explanations, making it subject to the widely observed phenomenon of reward hacking [Skalse et al., 2022; Singhal et al., 2023; Chen et al., 2024], where the aligned LLMs generate high-reward responses (rated by the RM) that do not align with actual human…”
- L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learning (2025, unknown): “The first aims primarily to reduce unnecessary verbosity (as often desired in RLHF-tuned instruction-following models) while the second aims either to impose maximum length budgets or achieve precise token-level length adherence (Jie et al., 2023; Yuan et al., 2024; Singhal et al., 2024).”