Can OpenAI's Codex Fix Bugs?: An evaluation on QuixBugs

By Julian Aron Prenner, Hlib Babii, Romain Robbes. 2022

165 citations recorded by the source corpus.

OpenAI's Codex, a GPT-3like model trained on a large code corpus, has made headlines in and outside of academia. Given a short user-provided description, it is capable of synthesizing code snippets that are syntactically and semantically valid in most cases. In this work, we want to investigate whether Codex is able to localize and fix bugs, two important tasks in automated program repair. Our initial evaluation uses the multi-language QuixBugs benchmark (40 bugs in both Python and Java). We find that, despite not being trained for APR, Codex is surprisingly effective, and competitive with recent state of the art techniques. Our results also show that Codex is more successful at repairing Python than Java, fixing 50% more bugs in Python.

Subfields: Information Systems, Software, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

In Software

In Natural Language Processing

Citations in the corpus

Code Llama: Open Foundation Models for Code (2023, background): “…refinement, translation (Rozière et al., 2020; 2021; Szafraniec et al., 2023) fixing bugs (Yasunaga & Liang, 2021; Zhang et al., 2022a; Prenner et al., 2022), fixing build errors (Tarlow et al., 2020) or generating unit tests (Tufano et al., 2020; Li et al., 2022; Chen et al., 2023a),…”
A Survey on Large Language Models for Code Generation (2024, unknown): “Furthermore, Prenner et al. [69] investigate Codex’s ability to detect and fix bugs, which are essential tasks for automated program repair.”
AutoCodeRover: Autonomous Program Improvement (2024, unknown): “However, obtaining buggy locations for a large project is an essential and challenging task in resolving real-life bug reports.”
Software Testing With Large Language Models: Survey, Landscape, and Vision (2023, unknown): “Horváth et al. [99] empirically study the impact of program representation and model architecture on the repair performance.”
Automated Program Repair in the Era of Large Pre-trained Language Models (2022, unknown): “The evaluation demonstrated the scaling effect of LLMs where the repair results can be improved by using larger models.”
An Analysis of the Automatic Bug Fixing Performance of ChatGPT (2023, methodology): “Benchmark problem ChatGPT Codex [15] CoCoNut [14] Standard APR [13]”