Learning to summarize from human feedback
By Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan J. Lowe, Chelsea Voss, Alec Radford, Dario Amodei, Paul Christiano. 2020
3,335 citations recorded by the source corpus.
As language models become more powerful, training and evaluation are increasingly bottlenecked by the data and metrics used for a particular task. For example, summarization models are often trained to predict human reference summaries and evaluated using ROUGE, but both of these metrics are rough proxies for what we really care about---summary quality. In this work, we show that it is possible to significantly improve summary quality by training a model to optimize for human preferences. We collect a large, high-quality dataset of human comparisons between summaries, train a model to predict the human-preferred summary, and use that model as a reward function to fine-tune a summarization policy using reinforcement learning. We apply our method to a version of the TL;DR dataset of Reddit posts and find that our models significantly outperform both human reference summaries and much larger models fine-tuned with supervised learning alone. Our models also transfer to CNN/DM news articles, producing summaries nearly as good as the human reference without any news-specific fine-tuning. We conduct extensive analyses to understand our human feedback dataset and fine-tuned models We establish that our reward model generalizes to new datasets, and that optimizing our reward model results in better summaries than optimizing ROUGE according to humans. We hope the evidence from our paper motivates machine learning researchers to pay closer attention to how their training loss affects the model behavior they actually want.
Subfields: Artificial Intelligence, Natural Language Processing, Alignment & RLHF
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Luke S. Zettlemoyer
- Percy Liang
- Yejin Choi
- Yoshua Bengio
- Graham Neubig
- Quoc V. Le
In Natural Language Processing
- Mohit Bansal
- Yejin Choi
- Luke S. Zettlemoyer
- Jianfeng Gao
- Hannaneh Hajishirzi
- Noah A. Smith
- Percy Liang
- Graham Neubig
In Alignment & RLHF
- Angelica Chen
- Geoffrey Irving
- Ethan Perez
- Richard Ngo
- Jan Leike
- Danqi Chen
- Tomasz Korbak
- Sergey Levine
Citations in the corpus
- Llama 2: Open Foundation and Fine-Tuned Chat Models (2023, methodology): “The method, first showcased by Stiennon et al. (2020) in the context of text-summarization tasks, has since been extended to a range of other applications.”
- Training language models to follow instructions with human feedback (2022, methodology): “Specifically, we use reinforcement learning from human feedback (RLHF; Christiano et al., 2017; Stiennon et al., 2020) to fine-tune GPT-3 to follow a broad class of written instructions (see Figure 2).”
- Constitutional AI: Harmlessness from AI Feedback (2022, methodology): “Simplicity and Transparency The widely used reinforcement learning from human feedback (RLHF) method [Christiano et al., 2017, Stiennon et al., 2020] for training more helpful, honest, and harmless AI systems [Bai et al., 2022, Thoppilan et al., 2022, Ouyang et al., 2022, Glaese et al., 2022]…”
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model (2023, methodology): “For summarization, automatic evaluation metrics such as ROUGE can be poorly correlated with human preferences [38], and prior work has found that fine-tuning LMs using PPO on human preferences to provide more effective summaries.”
- TruthfulQA: Measuring How Models Mimic Human Falsehoods (2021, background): “Models could be fine-tuned on a set of examples chosen to demonstrate truthfulness (Solaiman and Dennison, 2021) or fine-tuned by reinforcement learning from human feedback (Stiennon et al., 2020).”
- Scaling Laws for Transfer (2021, background): “One example of an expensive dataset is human preferences of what makes a good summary [SOW + 20].”
- Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “This effect is especially pronounced in CNN/DM where raters even prefer the first three sentences to the summary provided in the dataset (Stiennon et al., 2020).”
- OpenAssistant Conversations - Democratizing Large Language Model Alignment (2023, methodology): “In RLHF , the human acts as a teacher and provides feedback in the form of rewards or penalties.”
- Self-Refine: Iterative Refinement with Self-Feedback (2023, background): “RLHF (Stiennon et al., 2020) optimize for human preference trained on feedback single (human) (not self gen.”
- The Capacity for Moral Self-Correction in Large Language Models (2023, methodology): “We study decoder-only transformer models fine-tuned with Reinforcement Learning from Human Feedback (RLHF) [13, 57] to function as helpful dialogue models.”
- Augmented Language Models: a Survey (2023, background): “In such cases, RLHF is typically applied after an initial supervised fine-tuning phase using a small number of expert demonstrations for the corresponding downstream task (Ramamurthy et al., 2022; Ouyang et al., 2022; Stiennon et al., 2020).”
- Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, background): “, 2023); RL based fine-tuning (Alabdulkarim et al., 2021; Liu et al., 2021b; Ouyang et al., 2022; Stiennon et al., 2020); Prompt-based learning (Gehman et al.”
- A General Language Assistant as a Laboratory for Alignment (2021, methodology): “A similar approach applying human feedback to greatly improve the performance of language models on summary-writing had already been demonstrated [SOW + 20].”
- Recursively Summarizing Books with Human Feedback (2021, methodology): “For training the model, we most closely follow the procedure of Stiennon et al. (2020).”
- Alignment of Language Agents (2021, methodology): “Recent work (Stiennon et al., 2020) has preferred batch corrections rather than fully online corrections for practical reasons – thus there is a tradeoff between online error correction”
- Scaling Laws for Reward Model Overoptimization (2022, methodology): “Such reward models have been used to train language models to perform a variety of complex tasks that are hard to judge automatically, including summarization (Stiennon et al., 2020), question-answering (Nakano et al.”