Learning to summarize with human feedback

By Nisan Stiennon, Ouyang Long, Jeffrey Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, Paul Christiano. 2020

121 citations recorded by the source corpus.

Subfields: Alignment & RLHF

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Alignment & RLHF

Citations in the corpus

Alignment of Language Agents (2021, methodology): “An example in the language agent setting is on the task of summarization using deep RL from a learnt reward model based on human feedback data (Stiennon et al., 2020).”
Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models (2023, methodology): “Following the instruction of Transformer [39], pre-trained LLMs demonstrate a powerful text understanding and generation capability [31, 19, 37, 5], and such breakthroughs also beneﬁt the VL modelling [13, 14, 3, 49], where these works add an extra adapter modules [17] in the pre-trained LLMs to…”
RewardBench: Evaluating Reward Models for Language Modeling (2024, methodology): “Incorporating human feedback into models in this way has been used to improve summarization (Stiennon et al., 2020; Wu et al., 2021), question answering (Nakano et al., 2021), image models (Lee et al., 2023) and instruction following in general (Ouyang et al., 2022).”
A Survey on Hallucination in Large Vision-Language Models (2024, unknown): “This method effectively enhances the model robustness.”
Generating Sequences by Learning to Self-Correct (2022, background): “Scalar measures are often improved with reinforcement learning (RL) on a base generator (Ziegler et al., 2019; Stiennon et al., 2020; Lu et al., 2022a), which is infeasible for improving many language models (e.g. those accessed through an API), and uses only scalar feedback.”
Visual-RFT: Visual Reinforcement Fine-Tuning (2025, unknown): “While there has been significant research on using reinforcement learning to enhance LLMs during post-training [1, 3, 27, 30, 34, 35, 38, 44, 48, 62, 63], the progress for LVLMs has been slower.”
RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-Grained Correctional Human Feedback (2023, background): “[15] distinguishes the inaccurate parts in responses via human annotation, and internally discourages the halluci-nated parts by direct preference optimization.”
AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors in Agents (2023, unknown): “…variations): Words: ${task_description} # Previous Solution The paragraph you gave in the last step is: `` {former_solution}``` # are some critics on the above solution: `` {critic_opinions}``` Using the these information, provide a paragraph that contains all the given words or their variations.”