Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback

By Stephen Casper, Xander Davies, Claudia Shi, T. Gilbert, J'er'emy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro J Freire and 22 more. Trans. Mach. Learn. Res., 2023

934 citations recorded by the source corpus.

Reinforcement learning from human feedback (RLHF) is a technique for training AI systems to align with human goals. RLHF has emerged as the central method used to finetune state-of-the-art large language models (LLMs). Despite this popularity, there has been relatively little public work systematizing its flaws. In this paper, we (1) survey open problems and fundamental limitations of RLHF and related methods; (2) overview techniques to understand, improve, and complement RLHF in practice; and (3) propose auditing and disclosure standards to improve societal oversight of RLHF systems. Our work emphasizes the limitations of RLHF and highlights the importance of a multi-faceted approach to the development of safer AI systems.

Subfields: Software

View Paper: https://www.semanticscholar.org/paper/6eb46737bf0ef916a7f906ec6a8da82a45ffb623

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Software

Citations in the corpus

Rethinking machine unlearning for large language models (2024, neutral): “We aspire for this work to pave the way for developing LLM unlearning, illuminating its opportunities, challenges, and untapped potential.”
Reinforcement Learning from Automatic Feedback for High-Quality Unit Test Generation (2023, positive): “Policy models tended to diverge substantially from the base model in order to maximize reward from the reward model, resulting in errant behavior such as (1) mode collapse [11] where the model learned to generate a narrow band of tests (such as empty tests like TestFocalMethod(){}), and (2) catastro”
SimPO: Simple Preference Optimization with a Reference-Free Reward (2024, positive): “While the classical RLHF approach [58, 65] has shown impressive results, it presents optimization challenges due to its multi-stage procedure, which involves training a reward model and then optimizing a policy model to maximize that reward [13].”
Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint (2023, background): “We refer interested readers to Wirth et al. (2017); Casper et al. (2023) for a detailed survey but focus on the most related works here.”
Safe RLHF: Safe Reinforcement Learning from Human Feedback (2023, positive): “Because of the bias and noise in human feedback (Wu et al., 2023a), some methods optimizing on a sole preference may lead the model to some local optimal solution (Casper et al., 2023).”
Managing extreme AI risks amid rapid progress (2023, background): “These R&D challenges include: Oversight and honesty: More capable AI systems can better exploit weaknesses in technical over-sight and testing 42,46,47 —for example, by producing false but compelling output 43,48,49 .”
A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics (2023, neutral): “According to the definition from the study [257], RLHF refers to methods that combine three interconnected steps: feedback collection, reward modeling, and policy optimization”
Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study (2024, background): “There exist previous works that have studied these methods through hyper-parameter tuning and analyzed the effects of the quality reward model quality (Zheng et al., 2023; Casper et al., 2023).”
RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment (2023, neutral): “Assistant:”
Role play with large language models (2023, background): “The resulting guardrails can reduce a dialogue agent’s potential for harm, but can also attenuate a model’s expressivity and creativity 30 . branches, keeping track of nodes where a narrative diverges in interesting ways, revisiting alternative branches at leisure.”
Disentangling Length from Quality in Direct Preference Optimization (2024, background): “However, prior works have demonstrated that reward models can be exploited (Casper et al., 2023; Gao et al., 2023) due to a Goodhart’s law effect (Clark and Amodei, 2016; Manheim and Garrabrant, 2019; Skalse et al., 2022; Lambert and Calandra, 2023).”
Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs (2024, methodology): “Recent advancements in applying reinforcement learning techniques (Schulman et al., 2017), particularly Reinforcement Learning from Human Feedback (RLHF) (Bai et al., 2022; Casper et al., 2023), have shown substantial potential for enhancing LLMs.”
The Participatory Turn in AI Design: Theoretical Foundations and the Current State of Practice (2023, background): “[46] Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, et al.”
Model Alignment as Prospect Theoretic Optimization (2024, methodology): “However, human feedback is often discussed only in the context of preferences (e.g., output y w ≻ y l for input x ), even though it can take many forms (e.g., approval/disapproval of y given Proceedings of the 41 st International Conference on Machine Learning , Vienna, Austria.”
OpenChat: Advancing Open-source Language Models with Mixed-Quality Data (2023, methodology): “However, all existing RLHF methods require high-quality pairwise or ranking-based preference data for preference modeling, which inevitably require expensive human expert annotations (Casper et al., 2023).”
Large Language Model Alignment: A Survey (2023, methodology): “Although RL-based methods have been successfully applied to align LLMs to human preferences, they require reward modeling, a process potentially susceptible to misalignment and systemic imperfections (Casper et al., 2023).”
S-SPPO: Semantic-Calibrated Self-Play Preference Optimization (2026, unknown): “While effective, this pipeline is often criticized for its complexity, training instability, and sensitivity to hyperparameter tuning (Engstrom et al., 2020; Azar et al., 2024; Casper et al., 2023; Rafailov et al., 2023).”