Automated Program Repair via Conversation: Fixing 162 out of 337 Bugs for $0.42 Each using ChatGPT

By Chun Xia, Lingming Zhang. International Symposium on Software Testing and Analysis, 2023

348 citations recorded by the source corpus.

Automated Program Repair (APR) aims to automatically generate patches for buggy programs. Traditional APR techniques suffer from a lack of patch variety as they rely heavily on handcrafted or mined bug fixing patterns and cannot easily generalize to other bug/fix types. To address this limitation, recent APR work has been focused on leveraging modern Large Language Models (LLMs) to directly generate patches for APR. Such LLM-based APR tools work by first constructing an input prompt built using the original buggy code and then querying the LLM to either fill-in (cloze-style APR) the correct code at the bug location or to produce a completely new code snippet as the patch. While the LLM-based APR tools are able to achieve state-of-the-art results, they still follow the classic Generate and Validate (GV) repair paradigm of first generating lots of patches by sampling from the same initial prompt and then validating each one afterwards. This not only leads to many repeated patches that are incorrect, but also misses the crucial and yet previously ignored information in test failures as well as in plausible patches. To address these aforementioned limitations, we propose ChatRepair, the first fully automated conversation-driven APR approach that interleaves patch generation with instant feedback to perform APR in a conversational style. ChatRepair first feeds the LLM with relevant test failure information to start with, and then learns from both failures and successes of earlier patching attempts of the same bug for more powerful APR. For earlier patches that failed to pass all tests, we combine the incorrect patches with their corresponding relevant test failure information to construct a new prompt for the LLM to generate the next patch. In this way, we can avoid making the same mistakes. For earlier patches that passed all the tests (i.e., plausible patches), we further ask the LLM to generate alternative variations of the original plausible patches. In this way, we can further build on and learn from earlier successes to generate more plausible patches to increase the chance of having correct patches. While our approach is general, we implement ChatRepair using state-of-the-art dialogue-based LLM – ChatGPT. Our evaluation on the widely studied Defects4j dataset shows that ChatRepair is able to achieve the new state-of-the-art in repair performance, achieving 114 and 48 correct fixes on Defects4j 1.2 and 2.0 respectively. By calculating the cost of accessing ChatGPT, we can fix 162 out of 337 bugs for $0.42 each!

Subfields: Information Systems, Software

View Paper: https://www.semanticscholar.org/paper/bb607525252a3d0088b8f88804ed727bd3505b99

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

1. Yinlin Deng
2. Shin Yoo
3. Mark Harman
4. Chun Xia
5. Lingming Zhang
6. Ling-Ming Zhang
7. Soren Dunn
8. Michael Pradel

In Software

1. Yinlin Deng
2. Shin Yoo
3. Mark Harman
4. Chun Xia
5. Lingming Zhang
6. Ling-Ming Zhang
7. Soren Dunn
8. Michael Pradel

Citations in the corpus

• Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation (2023, background): “LLMs have demonstrated state-ofthe-art performance on various code-related tasks, including code generation/synthesis [11, 28, 24], program repair [58, 59, 57], code translation [27, 45] and code summarization [1, 30].”
• Fuzz4ALL: Universal Fuzzing with Large Language Models (2023, methodology): “The algorithm then moves on to sampling with higher temperature to obtain more diverse prompts (line 5), as done in prior work [13, 79].”
• A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly (2023, background): “While it does not directly protect data integrity, it identifies abnormal or suspicious behavior that can potentially compromise data integrity (as well as data confidentiality and data reliability).”
• BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions (2024, methodology): “To improve the program quality and ease human annotation, we propose a conversion-driven framework for program refactoring and test case generations inspired by [62].”
• Large Language Models for Software Engineering: Survey and Open Problems (2023, background): “They further proposed ChatRepair [149], showing that the conversational approach fixes 162 out of 337 bugs for only $0.”
• Agentless: Demystifying LLM-based Software Engineering Agents (2024, methodology): “We refer to this as the repository structure format , which begins with the root folder of the repository and organizes code files or folder names.”
• Large Language Models for Software Engineering: A Systematic Literature Review (2023, background): “Using LLMs to address these SE tasks has shown a wealth of potential breakthroughs [26, 30, 137, 253, 264, 300, 301, 329].”