Training Language Models with Natural Language Feedback

By J'er'emy Scheurer, Jon Ander Campos, Jun Shern Chan, Angelica Chen, Kyunghyun Cho, Ethan Perez. 2022

Subfields: Natural Language Processing, Alignment & RLHF

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Alignment & RLHF

Citations in the corpus

Self-Refine: Iterative Refinement with Self-Feedback (2023, background): “Leveraging human- and machine-generated natural language (NL) feedback for refining outputs has been effective for a variety of tasks, including summarization (Scheurer et al., 2022), script generation (Tandon et al.”
Self-critiquing models for assisting human evaluators (2022, methodology): “Ideally, models could use human natural language feedback to improve their own outputs [SCC22].”
Aligning Language Models with Preferences through f-divergence Minimization (2023, methodology): “Similar work involving model self-critique and natural language feedback includes (Zhao et al., 2021; Scheurer et al., 2022; Saunders et al., 2022) f -divergence objectives for generative models In the literature, there have been several studies exploring the use of f -divergences in generative…”
ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation (2023, background): “Researchers have explored RL for language models to achieve more truthful, helpful, and harmless outcomes [36, 2, 60, 54, 49, 27, 45, 3].”
DialGuide: Aligning Dialogue Model Behavior with Developer Guidelines (2022, background): “…work has explored editing models by computing targeted changes in the model’s parameters (Sinitsin et al., 2020; Hase et al., 2021; Mitchell et al., 2021; Meng et al., 2022), while others have explored natural language feedback (Madaan et al., 2022; Scheurer et al., 2022; Zeidler et al., 2022).”
Robust Preference Learning for Storytelling via Contrastive Reinforcement Learning (2022, background): “Prior work on preference learning for language models using natural language critiques include (Scheurer et al., 2022; Saunders et al., 2022).”
Fine-Grained Human Feedback Gives Better Rewards for Language Model Training (2023, methodology): “[5, 35, 34] use a refinement model to refine model outputs conditioning on NL human feedback and then use a reward model to select the best refined outputs for supervised fine-tuning.”
CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing (2023, methodology): “Natural Language Feedback The technique of using natural language (NL) feedback is widely adopted to improve various tasks (Rupprecht et al., 2018; Scheurer et al., 2022).”
OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement (2024, unknown): “For many sequence generation tasks, iterative approaches are often taken to improve the generation quality, e.g., script generation (Tandon et al., 2021), summarization (Scheurer et al., 2022), and other tasks as shown in (Madaan et al., 2022; Saunders et al., 2022).”