Fine-Tuning Language Models from Human Preferences
By Daniel M. Ziegler, Nisan Stiennon, Jeff Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, Geoffrey Irving. 2019
2,638 citations recorded by the source corpus.
Reward learning enables the application of reinforcement learning (RL) to tasks where reward is defined by human judgment, building a model of reward by asking humans questions. Most work on reward learning has used simulated environments, but complex information about values is often expressed in natural language, and we believe reward learning for language is a key to making RL practical and safe for real-world tasks. In this paper, we build on advances in generative pretraining of language models to apply reward learning to four natural language tasks: continuing text with positive sentiment or physically descriptive language, and summarization tasks on the TL;DR and CNN/Daily Mail datasets. For stylistic continuation we achieve good results with only 5,000 comparisons evaluated by humans. For summarization, models trained with 60,000 comparisons copy whole sentences from the input but skip irrelevant preamble; this leads to reasonable ROUGE scores and very good performance according to our human labelers, but may be exploiting the fact that labelers rely on simple heuristics.
Subfields: Artificial Intelligence, Large Language Models, Alignment & RLHF
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Yejin Choi
- Percy Liang
- Jianfeng Gao
- Luke S. Zettlemoyer
- Sergey Levine
- Yoshua Bengio
- Quoc V. Le
In Large Language Models
- Yejin Choi
- Noah A. Smith
- Luke S. Zettlemoyer
- Danqi Chen
- Hannaneh Hajishirzi
- Percy Liang
- Angelica Chen
- Hinrich Schütze
In Alignment & RLHF
- Angelica Chen
- Ethan Perez
- Geoffrey Irving
- Tomasz Korbak
- Yejin Choi
- Kyunghyun Cho
- J'er'emy Scheurer
- Danqi Chen
Citations in the corpus
- Training language models to follow instructions with human feedback (2022, methodology): “Our methodology follows that of Ziegler et al. (2019) and Stiennon et al. (2020), who applied it in the stylistic continuation and summarization domains.”
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model (2023, methodology): “…human judgments of response quality are often easier to collect than expert demonstrations, and thus subsequent works have fine-tuned LLMs with datasets of human preferences, improving proficiency in translation [18], summarization [38, 48], story-telling [48], and instruction-following [26, 32].”
- OpenAssistant Conversations - Democratizing Large Language Model Alignment (2023, background): “…fine-tuning of language models that has recently emerged as a promising approach to making large language models more in line with human preferences by generating more desirable outputs based on explicitly collected human preference data [6, 7, 8, 9, 10, 11] and thus making them more useful.”
- Large Language Models Are Human-Level Prompt Engineers (2022, background): “To answer this question and steer LLMs toward desired behaviors, recent work has considered fine-tuning (Ouyang et al., 2022; Ziegler et al., 2019), in-context learning (Brown et al., 2020), and several forms of prompt generation (Gao, 2021), including both differentiable tuning of soft prompts…”
- Learning to summarize from human feedback (2020, methodology): “Most directly related to our work is previous work using human feedback to train summarization models with RL [3, 73].”
- ColdGANs: Taming Language GANs with Cautious Sampling Strategies (2020, background): “To this end, Ziegler et al. [61] proposed to directly reward systems using human judgment.”
- Recursively Summarizing Books with Human Feedback (2021, methodology): “Could one obtain improved performance by doing RL more on-policy, by generating the summary trees on the fly, or by training the reward model online as in Ziegler et al. (2019)?”
- GeDi: Generative Discriminator Guided Sequence Generation (2020, background): “…two categories: training or finetuning a model directly for controllable generation (Chan et al., 2021; Madotto et al., 2020; Keskar et al., 2019; Ziegler et al., 2019; Rajani et al., 2019; Fan et al., 2018; Ficler and Goldberg, 2017; Yu et al., 2017; Hu et al., 2017) or using a discriminator to…”
- Plug and Play Language Models: A Simple Approach to Controlled Text Generation (2019, methodology): “Controlled generation Current methods for controlled text generation involve either fine-tuning existing models with Reinforcement Learning (RL) (Ziegler et al., 2019), training Generative Ad-versarial Networks (Yu et al., 2017), or training conditional generative models (Kikuchi et al., 2016;…”
- Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog (2019, methodology): “Since the posting of the preprint of this paper, Ziegler and colleagues have used explicit human feedback to improve the summarization and text continuation performance of a large-scale language model [60].”
- Training Language Models with Language Feedback (2022, methodology): “highly according to human preferences, or a predictive model thereof (Ziegler et al., 2019; Stiennon et al., 2020; Nakano et al., 2021; Ouyang et al., 2022).”
- Red Teaming Language Models with Language Models (2022, methodology): “…to generate test cases (Dinan et al., 2019; Nie et al., 2020; Ribeiro et al., 2020; Röttger et al., 2021; Xu et al., 2021b; Wallace et al., 2021) and/or detect failures (Dinan et al., 2019; Ziegler et al., 2019; Nie et al., 2020; Stiennon et al., 2020; Xu et al., 2021b; Wu et al., 2021a).”
- Aligning Language Models with Preferences through f-divergence Minimization (2023, methodology): “In particular Jaques et al. (2017; 2019); Ziegler et al. (2019); Stiennon et al. (2020) have applied variations of KL-control (Todorov, 2006b; Kappen et al., 2013) which adds a penalty term to the reward term so that the resulting policy does not deviate too much from the original one in terms of…”