Loose lips sink ships: Mitigating Length Bias in Reinforcement Learning from Human Feedback
By Wei Shen, Rui Zheng, Wenyu Zhan, Jun Zhao, Shihan Dou, Tao Gui, Qi Zhang, Xuanjing Huang. 2023
90 citations recorded by the source corpus.
Reinforcement learning from human feedback serves as a crucial bridge, aligning large language models with human and societal values. This alignment requires a vast corpus of human feedback to learn a reward model, which is subsequently used to finetune language models. However, we have identified that the reward model often finds shortcuts to bypass its intended objectives, misleadingly assuming that humans prefer longer responses. The emergence of length bias often induces the model to favor longer outputs, yet it doesn't equate to an increase in helpful information within these outputs. In this paper, we propose an innovative solution, applying the Product-of-Experts (PoE) technique to separate reward modeling from the influence of sequence length. In our framework, the main expert concentrates on understanding human intents, while the biased expert targets the identification and capture of length bias. To further enhance the learning of bias, we introduce perturbations into the bias-focused expert, disrupting the flow of semantic information. Experimental results validate the effectiveness of our approach, indicating that language model performance is improved, irrespective of sequence length.
Subfields: Artificial Intelligence, Information Systems, Alignment & RLHF, Reinforcement Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Percy Liang
- Tatsunori B. Hashimoto
- Bryan Catanzaro
- Stefano Ermon
- Tom Goldstein
- Jianfeng Gao
- Sergey Levine
- Mohit Bansal
In Information Systems
- Percy Liang
- Tianyi Zhou
- Tom Goldstein
- Tatsunori B. Hashimoto
- Bryan Catanzaro
- Stefano Ermon
- Yann Dubois
- Luke S. Zettlemoyer
In Alignment & RLHF
- Rafael Rafailov
- Percy Liang
- Tatsunori B. Hashimoto
- Chelsea Finn
- James Thorne
- Jiwoo Hong
- Noah Lee
- Ryan Park
In Reinforcement Learning
- Percy Liang
- Rafael Rafailov
- Jiuhai Chen
- Chen Zhu
- Tianyi Zhou
- Mohammad Shoeybi
- Tom Goldstein
- Lichang Chen
Citations in the corpus
- Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators (2024, methodology): “Other biases Length is a well-known bias of automated evaluators of chatbot LLMs but several others have been noted, including a bias of models towards their own outputs Zheng et al. (2023), or presence of lists Dubois et al. (2023).”
- ORPO: Monolithic Preference Optimization without Reference Model (2024, methodology): “We attribute this to empirical evidence of the instability and reward mismatch problem of RLHF (Rafailov et al., 2023; Gao et al., 2022; Shen et al., 2023) as the RLHF models were trained with RM-350M and assessed with RM-1.”
- ODIN: Disentangled Reward Mitigates Hacking in RLHF (2024, methodology): “Many offline alternatives have also shown promises for better learning from feedbacks, which includes SLiC-HF [Zhao et al., 2023], DPO [Rafailov et al., 2023], IPO [Azar et al., 2023], KTO [Ethayarajh et al., 2023], ReST [Gulcehre et al., 2023] and RSO [Liu et al., 2024].”
- Disentangling Length from Quality in Direct Preference Optimization (2024, background): “This effect is even observed in in strong propriety models, such as GPT4 (John Schulman et al., 2022), which is now frequently used to evaluate the performance of other LLMs (Dubois et al., 2024; Zheng et al., 2023; Zeng et al., 2023).”