Self-Edit: Fault-Aware Code Editor for Code Generation
By Kechi Zhang, Zhuo Li, Jia Li, Ge Li, Zhi Jin. 2023
184 citations recorded by the source corpus.
Large language models (LLMs) have demonstrated an impressive ability to generate codes on competitive programming tasks. However, with limited sample numbers, LLMs still suffer from poor accuracy. Inspired by the process of human programming, we propose a generate-and-edit approach named Self-Edit that utilizes execution results of the generated code from LLMs to improve the code quality on the competitive programming task. We execute the generated code on the example test case provided in the question and wrap execution results into a supplementary comment. Utilizing this comment as guidance, our fault-aware code editor is employed to correct errors in the generated code.We perform extensive evaluations across two competitive programming datasets with nine different LLMs. Compared to directly generating from LLMs, our approach can improve the average of pass@1 by 89% on APPS-dev, 31% on APPS-test, and 48% on HumanEval over nine popular code generation LLMs with parameter sizes ranging from 110M to 175B. Compared to other post-processing methods, our method demonstrates superior accuracy and efficiency.
Subfields: Artificial Intelligence, Information Systems, Software, Code Generation & Program Synthesis, Software Engineering
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Zhi Jin
- Jianfeng Gao
- Armando Solar-Lezama
- Shin Yoo
- Mark Harman
- Theo X. Olausson
- Jeevana Priya Inala
- Kechi Zhang
In Information Systems
- Shin Yoo
- Mark Harman
- Shubho Sengupta
- Mitya Lyubarskiy
- Jie M. Zhang
- Angela Fan
- Beliz Gokkaya
- Armando Solar-Lezama
In Software
- Shin Yoo
- Mark Harman
- Shubho Sengupta
- Mitya Lyubarskiy
- Jie M. Zhang
- Angela Fan
- Beliz Gokkaya
- Armando Solar-Lezama
In Code Generation & Program Synthesis
In Software Engineering
Citations in the corpus
- Large Language Models for Software Engineering: Survey and Open Problems (2023, background): “Evaluations have often relied on small programming problems from programming courses [86], synthetically generated problem sets [87], and online judging platforms such as Leetcode [28], [64], [88].”
- Structured Chain-of-Thought Prompting for Code Generation (2023, unknown): “Code generation aims to automatically generate a program that satisfies a given natural language requirement [13, 14, 38].”
- Demystifying GPT Self-Repair for Code Generation (2023, background): “[36] explore self-repair without natural language feedback on APPS [14] using a diverse range of fine-tuned models.”
- CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challenges (2024, methodology): “When occurring errors, this tool will offer error information to facilitate LLMs to fix bugs until programs are error-free, which has been proven to be effective by many existing works (Chen et al., 2022; Zhang et al., 2023b) to improve the accuracy of output programs.”
- AgentCoder: Multi-Agent-based Code Generation with Iterative Testing and Optimisation (2023, methodology): “Such benefit of multi-agent collaborations with LLMs has also been illustrated in other multi-agent systems [ Chen et al. , 2023a; Zhang et al. , 2023a].”