Teaching Large Language Models to Self-Debug
By Xinyun Chen, Maxwell Lin, Nathanael Scharli, Denny Zhou. 2023
1,232 citations recorded by the source corpus.
Large language models (LLMs) have achieved impressive performance on code generation. However, for complex programming tasks, generating the correct solution in one go becomes challenging, thus some prior works have designed program repair approaches to improve code generation performance. In this work, we propose Self-Debugging, which teaches a large language model to debug its predicted program via few-shot demonstrations. In particular, we demonstrate that Self-Debugging can teach the large language model to perform rubber duck debugging; i.e., without any human feedback on the code correctness or error messages, the model is able to identify its mistakes by investigating the execution results and explaining the generated code in natural language. Self-Debugging achieves the state-of-the-art performance on several code generation benchmarks, including the Spider dataset for text-to-SQL generation, TransCoder for C++-to-Python translation, and MBPP for text-to-Python generation. On the Spider benchmark where there are no unit tests to verify the correctness of predictions, Self-Debugging with code explanation consistently improves the baseline by 2-3%, and improves the prediction accuracy on problems of the hardest level by 9%. On TransCoder and MBPP where unit tests are available, Self-Debugging improves the baseline accuracy by up to 12%. Meanwhile, by leveraging feedback messages and reusing failed predictions, Self-Debugging notably improves sample efficiency, and can match or outperform baseline models that generate more than 10x candidate programs.
Subfields: Artificial Intelligence, Hardware and Architecture, Information Systems, Large Language Models, Code Generation & Program Synthesis
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Graham Neubig
- Mohit Bansal
- Jianfeng Gao
- Percy Liang
- Luke S. Zettlemoyer
- Quoc V. Le
- Caiming Xiong
- Zhiyuan Liu
In Hardware and Architecture
- Pushmeet Kohli
- Armando Solar-Lezama
- Bernardino Romera-Paredes
- Xinyun Chen
- Shin Yoo
- Koushik Sen
- Mark Harman
- William Yang Wang
In Information Systems
In Large Language Models
In Code Generation & Program Synthesis
- Graham Neubig
- Pengcheng Yin
- Pushmeet Kohli
- Armando Solar-Lezama
- Xinyun Chen
- Koushik Sen
- Mark Harman
- Zhi Jin
Citations in the corpus
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (2024, methodology): “The self-repair scenario is based on previous works that tested the self-repair capabilities of LLMs (Olausson et al., 2023; Shinn et al., 2023; Chen et al., 2023).”
- Software Testing With Large Language Models: Survey, Landscape, and Vision (2023, neutral): “Li et al. [87] employs the LLM in finding the failure-inducing test cases.”
- Executable Code Actions Elicit Better LLM Agents (2024, methodology): “CodeAct can use readily available Python packages for an expanded action space instead of hand-crafted task-specific tools (Yuan et al., 2023; Shen et al., 2023).”
- AgentCoder: Multi-Agent-based Code Generation with Iterative Testing and Optimisation (2023, background): “Such benefit of multi-agent collaborations with LLMs has also been illustrated in other multi-agent systems [ Chen et al. , 2023a; Zhang et al. , 2023a].”
- Explaining Competitive-Level Programming Solutions using LLMs (2023, methodology): “Chen et al. (2023) utilizes self-generated explanations as feedback to its self-debug.”
- Large Language Models as Tool Makers (2023, methodology): “The ability of LLMs to self-debug has been demonstrated effectively in recent research (Madaan et al., 2023; Chen et al., 2023c; Lu et al., 2023; Kim et al., 2023).”
- Automatic Prompt Optimization with "Gradient Descent" and Beam Search (2023, background): “Concurrent to our work, Chen et al. (2023) propose editing SQL-generation prompts based on output feedback.”
- CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challenges (2024, methodology): “• We propose C ODE A GENT that leverages tools to tackle repo-level code generation.”
- Aligning Large Language Models with Human: A Survey (2023, methodology): “Differently, Chen et al. (2023b) propose using ChatGPT to directly assess the quality of instructions by assigning scores.”
- Communicative Agents for Software Development (2023, background): “Rather than discarding incorrect code outright, humans typically analyze and investigate code execution results to identify and rectify implementation errors [8].”
- Demystifying GPT Self-Repair for Code Generation (2023, background): “[7] assess Codex’s ability to self-repair across a variety of tasks, in a framework that closely resembles that which we study in this work.”
- Gorilla: Large Language Model Connected with Massive APIs (2023, neutral): “Researchers have proposed an array of strategies to prompt LLMs to perform better in coding tasks, including in-context learning [44, 18, 7], task decomposition [17, 46], and self-debugging [8, 34].”
- OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement (2024, neutral): “In line with best practices outlined in recent studies (Liu et al., 2023; Chen et al., 2023), OpenCodeInterpreter ’s solutions are generated via greedy decoding.”
- Large Language Models as Optimizers (2023, neutral): “…reasoning (Shinn et al., 2023; Madaan et al., 2023) and code generation performance (Chen et al., 2023e; Olausson et al., 2023; Shinn et al., 2023; Chen et al., 2023b), dialogue applications (Nair et al., 2023; Madaan et al., 2023; Yuan et al., 2023), and so on (Kim et al., 2023; Wang et al.,…”
- Large Language Models for Software Engineering: Survey and Open Problems (2023, methodology): “Chen et al. [150] introduced SELF-DEBUGGING, which teaches an LLM to debug its predicted code via few-shot learning, SELF-DEBUGGING reports baseline accuracy improvements of up to 12%.”
- Cognitive Architectures for Language Agents (2023, background): “…et al., 2023; Yao et al., 2022b; Parisi et al., 2022; Tang et al., 2023b), and websites (Shi et al., 2017; Nakano et al., 2021; Yao et al., 2022a; Zhou et al., 2023b; Gur et al., 2023; Deng et al., 2023) as well as general code execution (Yang et al., 2023; Le et al., 2022; Ni et al., 2023).”