Less training, more repairing please: revisiting automated program repair via zero-shot learning

By Chun Xia, Lingming Zhang. ESEC/SIGSOFT FSE, 2022

298 citations recorded by the source corpus.

Due to the promising future of Automated Program Repair (APR), researchers have proposed various APR techniques, including heuristic-based, template-based, and constraint-based techniques. Among such classic APR techniques, template-based techniques have been widely recognized as state of the art. However, such template-based techniques require predefined templates to perform repair, and their effectiveness is thus limited. To this end, researchers have leveraged the recent advances in Deep Learning to further improve APR. Such learning-based techniques typically view APR as a Neural Machine Translation problem, using the buggy/fixed code snippets as the source/target languages for translation. In this way, such techniques heavily rely on large numbers of high-quality bug-fixing commits, which can be extremely costly/challenging to construct and may limit their edit variety and context representation. In this paper, we aim to revisit the learning-based APR problem, and propose AlphaRepair, the first cloze-style (or infilling-style) APR approach to directly leveraging large pre-trained code models for APR without any fine-tuning/retraining on historical bug fixes. Our main insight is instead of modeling what a repair edit should look like (i.e., a NMT task), we can directly predict what the correct code is based on the context information (i.e., a cloze or text infilling task). Although our approach is general and can be built on various pre-trained code models, we have implemented AlphaRepair as a practical multilingual APR tool based on the recent CodeBERT model. Our evaluation of AlphaRepair on the widely used Defects4J benchmark shows for the first time that learning-based APR without any history bug fixes can already outperform state-of-the-art APR techniques. We also studied the impact of different design choices and show that AlphaRepair performs even better on a newer version of Defects4J (2.0) with 3.3X more fixes than best performing baseline, indicating that AlphaRepair can potentially avoid the dataset-overfitting issue of existing techniques. Additionally, we demonstrate the multilingual repair ability of AlphaRepair by evaluating on the QuixBugs dataset where AlphaRepair achieved the state-of-the-art results on both Java and Python versions.

Subfields: Information Systems, Software

View Paper: https://www.semanticscholar.org/paper/e37155d21818513bd40d64ee212099aac82bd6f8

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

1. Yinlin Deng
2. Chun Xia
3. Lingming Zhang
4. Haoran Peng
5. Chenyuan Yang
6. Jiawei Liu
7. Yuyao Wang
8. Soren Dunn

In Software

1. Yinlin Deng
2. Chun Xia
3. Lingming Zhang
4. Haoran Peng
5. Chenyuan Yang
6. Jiawei Liu
7. Yuyao Wang
8. Soren Dunn

Citations in the corpus

• Automated Program Repair in the Era of Large Pre-trained Language Models (2022, unknown): “While learning-based APR tools have shown improvements in both the number and variety of bugs that can be fixed [21], [22], they are still restricted by their training data which may contain unrelated commits and only contain limited bug-fix types, which may not generalize to unseen bug types [26].”
• Repairing Bugs in Python Assignments Using Large Language Models (2022, unknown): “Wefollow the approach of recent work [16], [17] in framing program repair as a code generation task that can be tackled with an LLMC.”
• Large Language Models Are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Models (2022, methodology): “Besides generative models, inspired by recent work [73, 74] on infilling-style program repair (where LLMs generate correct patches by directly filling in the correct code given the context), we also leverage infilling models to perform mutations by replacing a small part of an input program with mas”
• Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation (2023, background): “LLMs have demonstrated state-ofthe-art performance on various code-related tasks, including code generation/synthesis [11, 28, 24], program repair [58, 59, 57], code translation [27, 45] and code summarization [1, 30].”