RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback

By Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, Abhinav Rastogi. International Conference on Machine Learning, 2023

683 citations recorded by the source corpus.

Reinforcement learning from human feedback (RLHF) has proven effective in aligning large language models (LLMs) with human preferences, but gathering high-quality preference labels is expensive. RL from AI Feedback (RLAIF), introduced in Bai et al., offers a promising alternative that trains the reward model (RM) on preferences generated by an off-the-shelf LLM. Across the tasks of summarization, helpful dialogue generation, and harmless dialogue generation, we show that RLAIF achieves comparable performance to RLHF. Furthermore, we take a step towards"self-improvement"by demonstrating that RLAIF can outperform a supervised fine-tuned baseline even when the AI labeler is the same size as the policy, or even the exact same checkpoint as the initial policy. Finally, we introduce direct-RLAIF (d-RLAIF) - a technique that circumvents RM training by obtaining rewards directly from an off-the-shelf LLM during RL, which achieves superior performance to canonical RLAIF. Our results suggest that RLAIF can achieve performance on-par with using human feedback, offering a potential solution to the scalability limitations of RLHF.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/600ff4c4ae9fc506c86673c5ecce4fa90803e987

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

ODIN: Disentangled Reward Mitigates Hacking in RLHF (2024, methodology): “…success of several epochal LLM systems [Schulman et al., 2022, OpenAI, 2023, Pichai, 2023, Anthropic, 2023, Google, 2023], and more diverse sources of preferences have been used to train the reward model [Bai et al., 2022, Lee et al., 2023] or provide feedbacks directly in RL [Liu et al., 2023].”
Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models (2024, background): “Lee et al. (2023) demonstrated that AI feedback can achieve comparable or superior performance to human feedback in RL fine-tuning.”
Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study (2024, background): “Concurrent efforts have been undertaken to avoid reward model overoptimization (Ram´e et al., 2024), facilitate alignment data generation (Lee et al., 2023; Yang et al., 2023), and implement resource-efficient RLHF systems (Yao et al., 2023; Santacroce et al., 2023).”
Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision (2023, methodology): “Constitutional AI (Bai et al., 2022b; Lee et al., 2023) leverages AI feedback to align language models, but still uses an initial RLHF phase.”
Diffusion Model Alignment Using Direct Preference Optimization (2023, methodology): “In LLMs, learning from AI feedback has emerged as a strong alternative to learning from human preferences [25].”
Statistical Rejection Sampling Improves Preference Optimization (2023, background): “L on similar tasks has been shown to be close to human raters (Lee et al., 2023; Shu et al., 2023).”
A Survey on Knowledge Distillation of Large Language Models (2024, methodology): “…DARWIN Series (Xie et al., 2023a) utilizes a semi self-instruct for instruction generation for science papers, which is then used to fine-tune an LLM. SciGLM (Zhang et al., 2024) proposes to train a scientific LLM, which prompts a teacher LLM to generate detailed answers for unlabelled…”