Automated Program Repair in the Era of Large Pre-trained Language Models

By Chun Xia, Lingming Zhang. International Conference on Software Engineering, 2022

380 citations recorded by the source corpus.

Automated Program Repair (APR) aims to help developers automatically patch software bugs. However, current state-of-the-art traditional and learning-based APR techniques face the problem of limited patch variety, failing to fix complicated bugs. This is mainly due to the reliance on bug-fixing datasets to craft fix templates (traditional) or directly predict potential patches (learning-based). Large Pre-Trained Language Models (LLMs), trained using billions of text/code tokens, can potentially help avoid this issue. Very recently, researchers have directly leveraged LLMs for APR without relying on any bug-fixing datasets. Meanwhile, such existing work either failed to include state-of-the-art LLMs or was not evaluated on realistic datasets. Thus, the true power of modern LLMs on the important APR problem is yet to be revealed. In this work, we perform the first extensive study on directly applying LLMs for APR. We select 9 recent state-of-the-art LLMs, including both generative and infilling models, ranging from 125M to 20B in size. We designed 3 different repair settings to evaluate the different ways we can use LLMs to generate patches: 1) generate the entire patch function, 2) fill in a chunk of code given the prefix and suffix 3) output a single line fix. We apply the LLMs under these repair settings on 5 datasets across 3 different languages and compare different LLMs in the number of bugs fixed, generation speed and compilation rate. We also compare the LLMs against recent state-of-the-art APR tools. Our study demonstrates that directly applying state-of-the-art LLMs can already substantially outperform all existing APR techniques on all our datasets. Among the studied LLMs, the scaling effect exists for APR where larger models tend to achieve better performance. Also, we show for the first time that suffix code after the buggy line (adopted in infilling-style APR) is important in not only generating more fixes but more patches with higher compilation rate. Besides patch generation, the LLMs consider correct patches to be more natural than other ones, and can even be leveraged for effective patch ranking or patch correctness checking. Lastly, we show that LLM-based APR can be further substantially boosted via: 1) increasing the sample size, and 2) incorporating fix template information.

Subfields: Information Systems, Software

View Paper: https://www.semanticscholar.org/paper/5aec1050fd62971f2b40b597ef7526fbc820beab

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

1. Yinlin Deng
2. Chun Xia
3. Soren Dunn
4. Lingming Zhang
5. Haoran Peng
6. Chenyuan Yang
7. Jiawei Liu
8. Yuyao Wang

In Software

1. Yinlin Deng
2. Chun Xia
3. Soren Dunn
4. Lingming Zhang
5. Haoran Peng
6. Chenyuan Yang
7. Jiawei Liu
8. Yuyao Wang

Citations in the corpus

• Large Language Models Are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Models (2022, methodology): “Besides generative models, inspired by recent work [74, 75] on in lling-style program repair (where LLMs generate correct patches by directly lling in the correct code given the context), we also leverage in lling models to perform mutations by replacing a small part of an input program with masked ”
• Can Transformers Learn to Solve Problems Recursively? (2023, background): “Once confined to the realm of symbolic methods, some of the most performant tools for synthesizing [3, 4, 21, 5], repairing [16, 35, 41], and even formally verifying [1, 42, 12, 37, 36, 13] programs now rest in part or in whole upon neural foundations.”
• RepairAgent: An Autonomous, LLM-Based Agent for Program Repair (2024, neutral): “The second and current generation of LLM-based repair introduces iterative approaches, which query the LLM repeatedly based on feedback obtained from previous fix attempts [19], [20], [21].”
• Agentless: Demystifying LLM-based Software Engineering Agents (2024, methodology): “We refer to this as the repository structure format , which begins with the root folder of the repository and organizes code files or folder names.”
• Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation (2023, background): “LLMs have demonstrated state-ofthe-art performance on various code-related tasks, including code generation/synthesis [11, 28, 24], program repair [58, 59, 57], code translation [27, 45] and code summarization [1, 30].”