Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
By Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova Dassarma, Dawn Drain, Stanislav Fort, Deep Ganguli, T. Henighan and 21 more. arXiv.org, 2022
4,254 citations recorded by the source corpus.
We apply preference modeling and reinforcement learning from human feedback (RLHF) to finetune language models to act as helpful and harmless assistants. We find this alignment training improves performance on almost all NLP evaluations, and is fully compatible with training for specialized skills such as python coding and summarization. We explore an iterated online mode of training, where preference models and RL policies are updated on a weekly cadence with fresh human feedback data, efficiently improving our datasets and models. Finally, we investigate the robustness of RLHF training, and identify a roughly linear relation between the RL reward and the square root of the KL divergence between the policy and its initialization. Alongside our main results, we perform peripheral analyses on calibration, competing objectives, and the use of OOD detection, compare our models with human writers, and provide samples from our models using prompts appearing in recent related work. Figure These plots show that PM accuracy decreases as we focus exclusively on comparisons between pairs of samples with high score. We have normalized all preference models to have the same mean score on a held-out dataset so that they’re directly comparable, and then plotted accuracy for the comparisons where both samples have scores above a specific threshold.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Alignment & RLHF, Reinforcement Learning
View Paper: https://arxiv.org/abs/2204.05862
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Luke S. Zettlemoyer
- Jianfeng Gao
- Percy Liang
- Graham Neubig
- Yejin Choi
- Quoc V. Le
- Yoshua Bengio
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Ming-Hsuan Yang
- Xiaogang Wang
- Jitendra Malik
- Andrew Zisserman
- Bernt Schiele
In Alignment & RLHF
In Reinforcement Learning
- Sergey Levine
- Pieter Abbeel
- Chelsea Finn
- Shixiang Shane Gu
- Yuxi Li
- Shimon Whiteson
- Ofir Nachum
- Jimmy Ba
Citations in the corpus
- Llama 2: Open Foundation and Fine-Tuned Chat Models (2023, methodology): “Others have found that helpfulness and safety sometimes trade off (Bai et al., 2022a), which can make it challenging for a single reward model to perform well on both.”
- Constitutional AI: Harmlessness from AI Feedback (2022, methodology): “All our RL runs used the same hyperparameters as our prior work [Bai et al., 2022].”
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model (2023, methodology): “| {z } This is the same objective optimized in prior works [48, 38, 1, 26] using the DPO-equivalent reward for the reward class of r � .”
- OpenAssistant Conversations - Democratizing Large Language Model Alignment (2023, background): “One possible solution is assistant-style fine-tuning of language models that has recently emerged as a promising approach to making large language models more in line with human preferences by generating more desirable outputs [3] and thus making them more useful.”
- Self-Refine: Iterative Refinement with Self-Feedback (2023, background): “…a variety of tasks, including summarization (Scheurer et al., 2022), script generation (Tandon et al., 2021), program synthesis (Le et al., 2022a; Yasunaga and Liang, 2020), and other tasks (Bai et al., 2022a; Schick et al., 2022b; Saunders et al., 2022a; Bai et al., 2022b; Welleck et al., 2022).”
- Towards Safer Generative Language Models: A Survey on Safety Risks, Evaluations, and Improvements (2023, methodology): “More recently, red team-ing is also a crucial step for large language models (Bai et al., 2022a; Touvron et al., 2023).”
- The Capacity for Moral Self-Correction in Large Language Models (2023, methodology): “This is likely due to fact that extended RLHF training tends to decrease the entropy of model outputs, which can lead to low sample diversity [3].”
- Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned (2022, methodology): “Many strategies have been developed to address some of these harms (e.g., [58, 4, 48, 36, 34, 19, 60]).”
- Judging LLM-as-a-judge with MT-Bench and Chatbot Arena (2023, background): “There has been a proliferation of LLM-based chat assistants (chatbots) that leverage supervised instruction fine-tuning and reinforcement learning with human feedback (RLHF) to unlock new instruction following and conversational abilities [27, 2, 26, 7, 43].”
- Measuring Progress on Scalable Oversight for Large Language Models (2022, methodology): “…two datasets (MMLU and time-limited QuALITY; Hendrycks et al., 2020; Pang et al., 2022) on which we expect our existing natural language assistant (Bai et al., 2022) to perform better than the participants could naïvely perform on their own, but on which we expect our assistant to nonetheless…”
- Scaling Laws for Reward Model Overoptimization (2022, methodology): “, 2018) has been used broadly in language models (Stiennon et al., 2020; Ouyang et al., 2022; Nakano et al., 2021; Bai et al., 2022).”
- Improving alignment of dialogue agents via targeted human judgements (2022, methodology): “In contrast to Ouyang et al. (2022), and in line with the findings in Bai et al. (2022), we measure negligible ‘alignment tax’ relative to DPC in few-shot evaluations on MMLU (see fig.”
- BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage (2022, background): “Alternatively, rather than learning from the conversation itself, one can augment the messaging system with a user interface that collects appropriate data, for example stack ranking potential responses (Ouyang et al., 2022; Bai et al., 2022).”