An Analysis of the Automatic Bug Fixing Performance of ChatGPT

By D. Sobania, Martin Briesch, Carol Hanna, J. Petke. 2023

382 citations recorded by the source corpus.

To support software developers in finding and fixing software bugs, several automated program repair techniques have been introduced. Given a test suite, standard methods usually either synthesize a repair, or navigate a search space of software edits to find test-suite passing variants. Recent program repair methods are based on deep learning approaches. One of these novel methods, which is not primarily intended for automated program repair, but is still suitable for it, is ChatGPT. The bug fixing performance of ChatGPT, however, is so far unclear. Therefore, in this paper we evaluate ChatGPT on the standard bug fixing benchmark set, QuixBugs, and compare the performance with the results of several other approaches reported in the literature. We find that ChatGPT's bug fixing performance is competitive to the common deep learning approaches CoCoNut and Codex and notably better than the results reported for the standard program repair approaches. In contrast to previous approaches, ChatGPT offers a dialogue system through which further information, e.g., the expected output for a certain input or an observed error message, can be entered. By providing such hints to ChatGPT, its success rate can be further increased, fixing 31 out of 40 bugs, outperforming state-of-the-art.

Subfields: Information Systems, Software, Large Language Models, Code Generation & Program Synthesis, Program Analysis & Verification

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

1. Jisun An
2. Jacob Andreas
3. Fan Huang
4. Haewoon Kwak
5. Bailin Wang
6. Najoung Kim

In Software

1. Jacob Andreas

In Large Language Models

1. Nikolaos D. Tselikas
2. Konstantinos I. Roumeliotis
3. Jacob Andreas
4. Najoung Kim

Citations in the corpus

• Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speech (2023, background): “most powerful text generation model, not surprisingly, researchers and practitioners have been evaluating its capability for various tasks, including question-and-answer in financial, medical, legal, and psychological areas [10], medical report simplification [13], bug fixing in computer programs [1”
• No More Manual Tests? Evaluating and Improving ChatGPT for Unit Test Generation (2023, background): “, program repair [52], [55], code generation [53].”
• Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks (2023, methodology): “…(Kung et al., 2023; Nori et al., 2023; Terwi-esch, 2023; i.a. ), exceed state-of-the-art methods on many traditional benchmarks (Sun et al., 2023; Sobania et al., 2023; Zhang et al., 2023a; Dhingra et al., 2023; i.a. ), and surpass human performance on tasks that require seemingly nontrivial…”
• Large Language Models for Software Engineering: A Systematic Literature Review (2023, background): “Using LLMs to address these SE tasks has shown a wealth of potential breakthroughs [26, 30, 137, 253, 264, 300, 301, 329].”
• Software Testing With Large Language Models: Survey, Landscape, and Vision (2023, neutral): “1 Unit test generation Unit Test Case Generation with Transformers and Focal Context 2021 [24] 2 Unit test generation CodeT: Code Generation with Generated Tests 2022 [25] 3 Unit test generation Interactive Code Generation via Test Driven User Intent Formalization 2022 [26] 4 Unit test generation A3”
• Summary of ChatGPT/GPT-4 Research and Perspective Towards the Future of Large Language Models (2023, background): “Sobania et al. [77] utilized ChatGPT for code bug fixing, and further improved the success rate of bug fixing by inputting more information through its dialogue system.”
• Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Models (2023, methodology): “[83] utilized ChatGPT for code bug fixing, and further improved the success rate of bug fixing by inputting more information through its dialogue system.”
• A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly (2023, background): “Anomaly detection is a key defense mechanism that identifies unusual behavior.”
• ChatGPT and Open-AI Models: A Preliminary Review (2023, background): “, 2023 investigated the efficacy of ChatGPT for automated bug fixing [53].”
• Exploring the Integration of Large Language Models in Industrial Test Maintenance Processes (2024, background): “As a result, LLMs have shown promise in automating or assisting with software engineering [127] and software testing [114] tasks, including test generation [97], documentation [42], refactoring [118], and automated program repair [104].”