Recursively Summarizing Books with Human Feedback
By Jeff Wu, Long Ouyang, Daniel M. Ziegler, Nissan Stiennon, Ryan J. Lowe, Jan Leike, Paul Christiano. 2021
371 citations recorded by the source corpus.
A major challenge for scaling machine learning is training models to perform tasks that are very difficult or time-consuming for humans to evaluate. We present progress on this problem on the task of abstractive summarization of entire fiction novels. Our method combines learning from human feedback with recursive task decomposition: we use models trained on smaller parts of the task to assist humans in giving feedback on the broader task. We collect a large volume of demonstrations and comparisons from human labelers, and fine-tune GPT-3 using behavioral cloning and reward modeling to do summarization recursively. At inference time, the model first summarizes small sections of the book and then recursively summarizes these summaries to produce a summary of the entire book. Our human labelers are able to supervise and evaluate the models quickly, despite not having read the entire books themselves. Our resulting model generates sensible summaries of entire books, even matching the quality of human-written summaries in a few cases ($\sim5\%$ of books). We achieve state-of-the-art results on the recent BookSum dataset for book-length summarization. A zero-shot question-answering model using these summaries achieves state-of-the-art results on the challenging NarrativeQA benchmark for answering questions about books and movie scripts. We release datasets of samples from our model.
Subfields: Artificial Intelligence, Natural Language Processing, Summarization
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Percy Liang
- Jianfeng Gao
- Luke S. Zettlemoyer
- Sergey Levine
- Quoc V. Le
- Yoshua Bengio
- Yejin Choi
In Natural Language Processing
- Mohit Bansal
- Yejin Choi
- Jianfeng Gao
- Luke S. Zettlemoyer
- Kyunghyun Cho
- Percy Liang
- Graham Neubig
- Noah A. Smith
In Summarization
- Ramakanth Pasunuru
- Christopher D. Manning
- K. McKeown
- Yejin Choi
- Tatsunori B. Hashimoto
- Hannaneh Hajishirzi
- Esin Durmus
- Bill Yuchen Lin
Citations in the corpus
- Training language models to follow instructions with human feedback (2022, methodology): “Similarly to Wu et al. (2021), we find that our SFT models overfit on validation loss after 1 epoch; however, we find that training for more epochs helps both the RM score and human preference ratings.”
- Show Your Work: Scratchpads for Intermediate Computation with Language Models (2021, neutral): “Wu et al. (2021) is an example of an approach in this spirit.”
- Finetuned Language Models Are Zero-Shot Learners (2021, methodology): “, 2021), summarization (Wu et al., 2021), as well as improved bias and fairness behavior (Solaiman & Dennison, 2021).”
- Augmented Language Models: a Survey (2023, positive): “…learning to use external tools, RLHF has also proven useful for a wide range of language generation tasks, from summarization (Ziegler et al., 2019; Wu et al., 2021; Stiennon et al., 2020) to training more helpful, harmless, and accurate assistants (Glaese et al., 2022; Cohen et al., 2022; Ouyang…”
- Measuring Progress on Scalable Oversight for Large Language Models (2022, background): “• Self-Critique, Amplification and Recursive Reward Modeling: In this family of approaches (Leike et al., 2018; Christiano et al., 2018; Wu et al., 2021; Saunders et al., 2022), loosely speaking, the human participants iteratively (i) supervise a model using a human-supervision-based technique like…”
- Improving alignment of dialogue agents via targeted human judgements (2022, methodology): “, 2019) and recursively summarising books (Wu et al., 2021), which simplifies the rating task from evaluating book-length summaries to passagelength summaries.”
- Self-critiquing models for assisting human evaluators (2022, methodology): “[WOZ21] implements a variant of recursive reward modeling [LKE18] on summarization tasks.”
- Red Teaming Language Models with Language Models (2022, methodology): “…to generate test cases (Dinan et al., 2019; Nie et al., 2020; Ribeiro et al., 2020; Röttger et al., 2021; Xu et al., 2021b; Wallace et al., 2021) and/or detect failures (Dinan et al., 2019; Ziegler et al., 2019; Nie et al., 2020; Stiennon et al., 2020; Xu et al., 2021b; Wu et al., 2021a).”
- Is Reinforcement Learning (Not) for Natural Language Processing?: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization (2022, methodology): “…et al., 2019), SS for Transformers (Mihaylova & Martins, 2019), Diffential SS (Goyal et al., 2017), LOLS (Lampouras & Vlachos, 2016; Chang et al., 2015), TextGAIL (Wu et al., 2021b), and SEARNN (Leblond et al., 2017), have been inspired by DAGGER (Ross et al., 2011) and SEARN (Daumé et al., 2009).”
- Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022, methodology): “In particular, we use CLIP (Radford et al. 2021) as the VLM (for egocentric perception, ViTL/14 with 428M params); Wav2CLIP (Wu et al. 2021a) as the sound-critic ALM and Google Cloud Speech-to-text API as the speech-to-text ALM; and GPT-3 with 175B params (Brown et al. 2020; Ouyang et al. 2022) and…”
- RRHF: Rank Responses to Align Language Models with Human Feedback without tears (2023, background): “Reinforcement learning techniques have been explored to align language models with human preferences [2, 4, 28, 13, 34, 21, 24].”
- Teaching Models to Express Their Uncertainty in Words (2022, methodology): “While we finetuned models using supervised learning, future work could explore the more flexible approach of reinforcement learning (Stiennon et al., 2020; Wu et al., 2021).”
- Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals (2022, methodology): “04359 [120] Jeff Wu, Long Ouyang, Daniel M.”
- Directly Fine-Tuning Diffusion Models on Differentiable Rewards (2023, positive): “…from human feedback (RLHF) have become widely used for fine-tuning large language models so they improve at summarization (Stiennon et al., 2020; Wu et al., 2021), instruction following (Ouyang et al., 2022), or general dialogue (Askell et al., 2021; Bai et al., 2022a; Glaese et al., 2022; Bai…”
- MemGPT: Towards LLMs as Operating Systems (2023, background): “Resursive summarization (Wu et al., 2021b) is a simple way to address overflowing context windows, however, recursive summarization is inherently lossy and eventually leads to large holes in the memory of the system (as we demonstrate in Section 3).”
- Reinforced Self-Training (ReST) for Language Modeling (2023, background): “Reinforcement learning from human feedback (RLHF) aims to address the alignment problem by using human preferences (Glaese et al., 2022; Stiennon et al., 2020; Wu et al., 2021).”
- A Survey of Large Language Models (2023, positive): “In early work [212, 215], human labelers often evaluate model-generated outputs in a coarsegrained manner (i.”
- Benchmarking Large Language Models for News Summarization (2023, methodology): “3 following prior work (Wu et al., 2021).”
- RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment (2023, methodology): “In addition to (2), another commonly adopted strategy to alleviate the overfitting problem is to introduce a KL penalty (as done by PPO [45]) between the current model and a reference model, usually taken as the initial pre-trained one.”