Teaching Large Language Models to Self-Debug

By Xinyun Chen, Maxwell Lin, Nathanael Scharli, Denny Zhou. 2023

1,232 citations recorded by the source corpus.

Large language models (LLMs) have achieved impressive performance on code generation. However, for complex programming tasks, generating the correct solution in one go becomes challenging, thus some prior works have designed program repair approaches to improve code generation performance. In this work, we propose Self-Debugging, which teaches a large language model to debug its predicted program via few-shot demonstrations. In particular, we demonstrate that Self-Debugging can teach the large language model to perform rubber duck debugging; i.e., without any human feedback on the code correctness or error messages, the model is able to identify its mistakes by investigating the execution results and explaining the generated code in natural language. Self-Debugging achieves the state-of-the-art performance on several code generation benchmarks, including the Spider dataset for text-to-SQL generation, TransCoder for C++-to-Python translation, and MBPP for text-to-Python generation. On the Spider benchmark where there are no unit tests to verify the correctness of predictions, Self-Debugging with code explanation consistently improves the baseline by 2-3%, and improves the prediction accuracy on problems of the hardest level by 9%. On TransCoder and MBPP where unit tests are available, Self-Debugging improves the baseline accuracy by up to 12%. Meanwhile, by leveraging feedback messages and reusing failed predictions, Self-Debugging notably improves sample efficiency, and can match or outperform baseline models that generate more than 10x candidate programs.

Subfields: Artificial Intelligence, Hardware and Architecture, Information Systems, Large Language Models, Code Generation & Program Synthesis

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Graham Neubig
  2. Mohit Bansal
  3. Jianfeng Gao
  4. Percy Liang
  5. Luke S. Zettlemoyer
  6. Quoc V. Le
  7. Caiming Xiong
  8. Zhiyuan Liu

In Hardware and Architecture

  1. Pushmeet Kohli
  2. Armando Solar-Lezama
  3. Bernardino Romera-Paredes
  4. Xinyun Chen
  5. Shin Yoo
  6. Koushik Sen
  7. Mark Harman
  8. William Yang Wang

In Information Systems

  1. Koushik Sen
  2. Graham Neubig
  3. Mark Harman
  4. Shin Yoo
  5. Pushmeet Kohli
  6. Xinyun Chen
  7. Pengfei Liu
  8. Maosong Sun

In Large Language Models

  1. Zhiyuan Liu
  2. Jianfeng Gao
  3. Maosong Sun
  4. Xin Jiang
  5. Qun Liu
  6. Graham Neubig
  7. Lifeng Shang
  8. Xinyun Chen

In Code Generation & Program Synthesis

  1. Graham Neubig
  2. Pengcheng Yin
  3. Pushmeet Kohli
  4. Armando Solar-Lezama
  5. Xinyun Chen
  6. Koushik Sen
  7. Mark Harman
  8. Zhi Jin

Citations in the corpus