RRHF: Rank Responses to Align Language Models with Human Feedback without tears
By Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, Feiran Huang. 2023
545 citations recorded by the source corpus.
Reinforcement Learning from Human Feedback (RLHF) facilitates the alignment of large language models with human preferences, significantly enhancing the quality of interactions between humans and models. InstructGPT implements RLHF through several stages, including Supervised Fine-Tuning (SFT), reward model training, and Proximal Policy Optimization (PPO). However, PPO is sensitive to hyperparameters and requires multiple models in its standard implementation, making it hard to train and scale up to larger parameter counts. In contrast, we propose a novel learning paradigm called RRHF, which scores sampled responses from different sources via a logarithm of conditional probabilities and learns to align these probabilities with human preferences through ranking loss. RRHF can leverage sampled responses from various sources including the model responses from itself, other large language model responses, and human expert responses to learn to rank them. RRHF only needs 1 to 2 models during tuning and can efficiently align language models with human preferences robustly without complex hyperparameter tuning. Additionally, RRHF can be considered an extension of SFT and reward model training while being simpler than PPO in terms of coding, model counts, and hyperparameters. We evaluate RRHF on the Helpful and Harmless dataset, demonstrating comparable alignment performance with PPO by reward model score and human labeling. Extensive experiments show that the performance of RRHF is highly related to sampling quality which suggests RRHF is a best-of-n learner. Codes available at https://github.com/GanjinZero/RRHF.
Subfields: Artificial Intelligence, Alignment & RLHF
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Luke S. Zettlemoyer
- Hannaneh Hajishirzi
- Yejin Choi
- Noah A. Smith
- Graham Neubig
- Percy Liang
In Alignment & RLHF
- Rafael Rafailov
- Dylan Hadfield-Menell
- Mari Ostendorf
- Noah A. Smith
- David Krueger
- Prithviraj Ammanabrolu
- J'er'emy Scheurer
- Yushi Hu
Citations in the corpus
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (2024, background): “It was noted that RL enhances Maj@K but not Pass@K. performance of SFT models can be improved through a series of preference alignment strategies (Song et al., 2023; Wang et al., 2023a; Yuan et al., 2023b).”
- Statistical Rejection Sampling Improves Preference Optimization (2023, methodology): “Recent works have proposed theoretically-justified approaches to optimizing relative preferences without relying on RL (Zhao et al., 2023; Yuan et al., 2023; Rafailov et al., 2023).”
- OpenChat: Advancing Open-source Language Models with Mixed-Quality Data (2023, methodology): “Taking the most popular SFT dataset ShareGPT used in Vicuna (Chiang et al., 2023) as an example, the distinct data sources from GPT-4 and GPT-3.”
- Large Language Model Unlearning (2023, unknown): “The current mainstream approach is RLHF (reinforcement learning from human feedback, and its variants) [Ouyang et al., 2022, Bai et al., 2022, Christiano et al., 2017, Yuan et al., 2023].”
- Preference Ranking Optimization for Human Alignment (2023, background): “PRO vs. RRHF , both of which are trained on HH-RLHF raw .”
- Safe RLHF: Safe Reinforcement Learning from Human Feedback (2023, unknown): “RLHF enables LLMs to progress towards more diverse goals by learning from human feedback (Ouyang et al., 2022; Yuan et al., 2023; Rafailov et al., 2023; Song et al., 2023; Yang et al., 2023).”
- Diffusion Model Alignment Using Direct Preference Optimization (2023, background): “Methods that fine-tune the policy model directly on feedback data [2, 12], or utilize a ranking loss on preference data to directly train the policy model [36, 52, 60, 62] have emerged.”
- Fine-Grained Human Feedback Gives Better Rewards for Language Model Training (2023, background): “[44] trains a reward model that predicts a single score for each model output and selects samples with the highest reward scores for supervised fine-tuning.”
- Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study (2024, methodology): “In contrast, reward-free methods, including Direct Preference Optimization (DPO) (Rafailov et al., 2023), RRHF (Yuan et al., 2023), and PRO (Song et al., 2023), eliminate the explicit use of a reward function.”
- HybridFlow: A Flexible and Efficient RLHF Framework (2024, unknown): “We perform RLHF on "Dahoas/ful-hh-rlhf" dataset [7] of HuggingFace, which is widely used for LLM alignment [64, 85].”
- WizardLM: Empowering Large Pre-Trained Language Models to Follow Complex Instructions (2023, unknown): “The whole annotating process is extremely expensive and time-consuming (Kopf et al., 2023; Chen et al., 2023; Sun et al., 2023; Yuan et al., 2023).”
- Aligning Large Language Models with Human: A Survey (2023, methodology): “Yuan et al. (2023) pro-pose RRHF, which further optimizes LLaMA-7B to align with human preferences using a similar framework described above.”
- A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly (2023, background): “Jin et al. [121] conducted a comprehensive study on LLMs’ understanding of binary code semantics [122] across different architectures and optimization levels, providing key insights for future research in this area.”
- Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (2023, methodology): “A different family of methods augments the language modeling objective to utilize feedback provided by the reward model (Korbak et al., 2023; Yuan et al., 2023; Rafailov et al., 2023).”
- RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment (2023, background): “In comparison, RRHF involves a diverse range of sources to generate data, and then finetune the model on the high-reward subset of these collected samples, while our primary focus lies in the online generated samples of the trained model itself, consistent with the setup of RL, where the behavior po”
- WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct (2023, unknown): “…into reinforcement learning pipelines and employed in rejection sampling to align Large Language Models (LLMs) with human preferences (Shen et al., 2021; Bai et al., 2022; Yuan et al., 2023b; Dong et al., 2023; Song et al., 2023; Touvron et al., 2023b; Rafailov et al., 2024; Meng et al., 2024).”
- SimPO: Simple Preference Optimization with a Reference-Free Reward (2024, unknown): “7 RRHF [86] and SLiC-HF [91] are ranking losses.”