Training Language Models with Language Feedback at Scale

By J'er'emy Scheurer, Jon Ander Campos, Tomasz Korbak, Jun Shern Chan, Angelica Chen, Kyunghyun Cho, Ethan Perez. 2023

138 citations recorded by the source corpus.

Pretrained language models often generate outputs that are not in line with human preferences, such as harmful text or factually incorrect summaries. Recent work approaches the above issues by learning from a simple form of human feedback: comparisons between pairs of model-generated outputs. However, comparison feedback only conveys limited information about human preferences. In this paper, we introduce Imitation learning from Language Feedback (ILF), a new approach that utilizes more informative language feedback. ILF consists of three steps that are applied iteratively: first, conditioning the language model on the input, an initial LM output, and feedback to generate refinements. Second, selecting the refinement incorporating the most feedback. Third, finetuning the language model to maximize the likelihood of the chosen refinement given the input. We show theoretically that ILF can be viewed as Bayesian Inference, similar to Reinforcement Learning from human feedback. We evaluate ILF's effectiveness on a carefully-controlled toy task and a realistic summarization task. Our experiments demonstrate that large language models accurately incorporate feedback and that finetuning with ILF scales well with the dataset size, even outperforming finetuning on human summaries. Learning from both language and comparison feedback outperforms learning from each alone, achieving human-level summarization performance.

Subfields: Artificial Intelligence, Large Language Models, Alignment & RLHF

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Alignment & RLHF

Citations in the corpus

AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback (2023, methodology): “However, there exist methods in the literature other than those explored in AlpacaFarm that can incorporate alternative sources of feedback such as natural language [74, 32, 22, 59, 55, 13, 56, 40], numeric ratings [44, 31], or execution traces [13].”
RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment (2023, methodology): “In language foundation model training [26, 39, 25, 2, 3, 14, 47, 45, 33], alignment is often achieved by Reinforcement Learning from Human Feedback (RLHF).”
Improving Code Generation by Training with Natural Language Feedback (2023, methodology): “Similar to our results on code generation, Scheurer et al. (2023) shows that ILF outperforms all supervised ﬁne-tuning baselines on text summarization.”
Understanding the Effects of RLHF on LLM Generalisation and Diversity (2023, background): “…reward function or feedback function and then trains a goal-conditioned policy on these outputs (similar to (Andrychowicz et al., 2017)); and ILF (Scheurer et al., 2023), which uses natural language human feedback to prompt the model to produce better outputs than its original inputs, and then…”
A Survey on Knowledge Distillation of Large Language Models (2024, methodology): “…labor-intensive, thus methods that learn from AI feedback are also proposed to align with human preferences (Bai et al., 2022a; Kwon et al., 2023; Scheurer et al., 2023; Kim et al., 2023a; Roit et al., 2023; Yang et al., 2024a; Lee et al., 2023a; Tunstall et al., 2023; Cui et al., 2023a; Wang et…”
High-Dimension Human Value Representation in Large Language Models (2024, background): “Numerous efforts have been made to imbue AI systems with ethical principles and moral values, from designing robust frameworks for value alignment [5, 6, 7] to incorporating diverse perspectives into training data [8, 9, 10, 11, 12].”