Training Language Models with Language Feedback

By J'er'emy Scheurer, Jon Ander Campos, Jun Shern Chan, Angelica Chen, Kyunghyun Cho, Ethan Perez. 2022

58 citations recorded by the source corpus.

Pretrained language models often do not perform tasks in ways that are in line with our preferences, e.g., generating offensive text or factually incorrect summaries. Recent work approaches the above issue by learning from a simple form of human evaluation: comparisons between pairs of model-generated task outputs. Comparison feedback conveys limited information about human preferences per human evaluation. Here, we propose to learn from natural language feedback, which conveys more information per human evaluation. We learn from language feedback on model outputs using a three-step learning algorithm. First, we condition the language model on the initial output and feedback to generate many refinements. Second, we choose the refinement with the highest similarity to the feedback. Third, we finetune a language model to maximize the likelihood of the chosen refinement given the input. In synthetic experiments, we first evaluate whether language models accurately incorporate feedback to produce refinements, finding that only large language models (175B parameters) do so. Using only 100 samples of human-written feedback, our learning algorithm finetunes a GPT-3 model to roughly human-level summarization ability.

Subfields: Natural Language Processing, Alignment & RLHF

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Alignment & RLHF

Citations in the corpus

Pretraining Language Models with Human Preferences (2023, methodology): “, 2022a) or supervised finetuning on data generated by other LMs (Scheurer et al., 2022) or highly curated collections of tasks phrased as instructions (Sanh et al.”
When Life Gives You Lemons, Make Cherryade: Converting Feedback from Bad Responses into Good Labels (2022, methodology): “Scheurer et al. (2022) ﬁne-tuned GPT-3 and InstructGPT on 100 examples of free-form textual feedback from humans to improve summarization tasks and found that only the larger models such as GPT-3 (175B) (Brown et al., 2020) can generate accurate reﬁnements using feedback.”
Language Model Cascades (2022, background): “Scheurer et al. (2022) incorporates natural language feedback, and ﬁnds that learning is signiﬁcantly more sample efﬁcient.”
Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs (2022, background): “…interactive learning: up-dating models trained on static text using super-vision either from humans (Stiennon et al., 2020; Ouyang et al., 2022; Scheurer et al., 2022) or from proxies for human behavior or social environments (Ammanabrolu et al., 2022a,b) based on broad coverage LLMs (Perez et…”
Training Language Models with Language Feedback at Scale (2023, methodology): “For a more detailed comparison to Scheurer et al. (2022) we refer to App.”
Aligning Text-to-Image Models using Human Feedback (2023, background): “…Ibarz et al., 2018; Lee et al., 2021, inter alia), and to more truthful and harmless instruction following and dialogue (Ouyang et al., 2022; Bai et al., 2022a; Ziegler et al., 2019; Wu et al., 2021; Stiennon et al., 2020; Liu et al., 2023; Scheurer et al., 2022; Bai et al., 2022b, inter alia).”
Improving Code Generation by Training with Natural Language Feedback (2023, methodology): “To train LLMs with language feedback, we propose an al-gorithm called Imitation learning from Language Feedback (ILF; Algorithm 1), which extends the work of Scheurer et al. (2022), who study the impact of learning from language feedback on text summarization models.”
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (2023, background): “…al., 2022, Ibarz et al., 2018, Stiennon et al., 2020, Chung et al., 2022, Bıyık et al., 2022, Zhou et al., 2023], or generating good demonstrations using an LLM, e.g., after conditioning human feedback provided in natural language [Scheurer et al., 2022, 2023, Chen et al., 2023, Xu et al., 2023b].”