Improving Code Generation by Training with Natural Language Feedback

By Angelica Chen. 2023

97 citations recorded by the source corpus.

The potential for pre-trained large language models (LLMs) to use natural language feedback at inference time has been an exciting recent development. We build upon this observation by formalizing an algorithm for learning from natural language feedback at training time instead, which we call Imitation learning from Language Feedback (ILF). ILF requires only a small amount of human-written feedback during training and does not require the same feedback at test time, making it both user-friendly and sample-efficient. We further show that ILF can be seen as a form of minimizing the KL divergence to the ground truth distribution and demonstrate a proof-of-concept on a neural program synthesis task. We use ILF to improve a Codegen-Mono 6.1B model's pass@1 rate by 38% relative (and 10% absolute) on the Mostly Basic Python Problems (MBPP) benchmark, outperforming both fine-tuning on MBPP and fine-tuning on repaired programs written by humans. Overall, our results suggest that learning from human-written natural language feedback is both more effective and sample-efficient than training exclusively on demonstrations for improving an LLM's performance on code generation tasks.

Subfields: Artificial Intelligence, Information Systems, Alignment & RLHF, Code Generation & Program Synthesis

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Alignment & RLHF

In Code Generation & Program Synthesis

Citations in the corpus

Teaching Large Language Models to Self-Debug (2023, methodology): “Different from prior works on utilizing human feedback for code repair, where the feedback message explains the code errors and how to fix them [6, 2], SELF-DEBUGGING teaches the model to identify the implementation errors via code explanation.”
Large Language Models for Software Engineering: A Systematic Literature Review (2023, background): “Open-source datasets [33, 140, 329, 379] refer to publicly accessible collections of data that are often disseminated through open-source platforms or repositories.”
Training Language Models with Language Feedback at Scale (2023, background): “In our concurrent paper (Chen et al., 2023), we show that ILF also achieves strong performance on code generation.”
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (2023, methodology): “Korbak et al. [2023] argue that it can be more effective to use human feedback during pretraining by using a reward model to filter, weight, or annotate pretraining data.”
AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback (2023, methodology): “However, there exist methods in the literature other than those explored in AlpacaFarm that can incorporate alternative sources of feedback such as natural language [74, 32, 22, 59, 55, 13, 56, 40], numeric ratings [44, 31], or execution traces [13].”
Fine-Grained Human Feedback Gives Better Rewards for Language Model Training (2023, methodology): “[5, 35, 34] use a refinement model to refine model outputs conditioning on NL human feedback and then use a reward model to select the best refined outputs for supervised fine-tuning.”
Large Language Models as Optimizers (2023, background): “…reasoning (Shinn et al., 2023; Madaan et al., 2023) and code generation performance (Chen et al., 2023e; Olausson et al., 2023; Shinn et al., 2023; Chen et al., 2023b), dialogue applications (Nair et al., 2023; Madaan et al., 2023; Yuan et al., 2023), and so on (Kim et al., 2023; Wang et al.,…”