QuixBugs: a multi-lingual program repair benchmark set based on the quixey challenge
By Derrick Lin, James Koppel, Armando Solar-Lezama. 2017
324 citations recorded by the source corpus.
Recent years have seen an explosion of work in automated program repair. While previous work has focused exclusively on tools for single languages, recent work in multi-language transformation has opened the door for multi-language program repair tools. Evaluating the performance of such a tool requires having a benchmark set of similar buggy programs in different languages. We present QuixBugs, consisting of 40 programs translated to both Python and Java, each with a bug on a single line. The QuixBugs benchmark suite is based on problems from the Quixey Challenge, where programmers were given a short buggy program and 1 minute to fix the bug.
Subfields: Information Systems, Software, Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Information Systems
- Chun Xia
- Lingming Zhang
- Neel Sundaresan
- Alexey Svyatkovskiy
- Dawn Drain
- Michele Tufano
- Shuai Lu
- Romain Robbes
In Software
- Chun Xia
- Lingming Zhang
- Neel Sundaresan
- Alexey Svyatkovskiy
- Dawn Drain
- Michele Tufano
- Shuai Lu
- Lin Tan
In Evaluation & Benchmarks
- Chun Xia
- Neel Sundaresan
- Romain Robbes
- Dawn Drain
- Julian Aron Prenner
- Catherine Olsson
- Shuai Lu
- Lingming Zhang
Citations in the corpus
- A General Language Assistant as a Laboratory for Alignment (2021, methodology): “We display results for two very similar python coding evaluations, the Codex HumanEval [CTJ + 21] and the QuixBugs challenge reformulated as a function synthesis task [LKCSL17] in figure 10.”
- Automated Program Repair in the Era of Large Pre-trained Language Models (2022, unknown): “2) QuixBugs-Python and -Java [63]: A multi-lingual repair benchmark with 40 classic programming problems.”
- Less training, more repairing please: revisiting automated program repair via zero-shot learning (2022, methodology): “Additionally, we also evaluate multilingual repair capability of AlphaRepair by testing on the Python version of QuixBugs.”
- InferFix: End-to-End Program Repair with LLMs (2023, methodology): “While a few bug datasets are already available, such as Defects4j [16], QuixBugs [20], ManySStuBs4J [17], UnifiedBug-Dataset [11] and many others, the dataset we introduce is differentiated by the amount and quality of information provided about each bug by the static analysis.”
- Can OpenAI's Codex Fix Bugs?: An evaluation on QuixBugs (2022, methodology): “In all of our experiments we rely on QuixBugs [16], a benchmark of 40 buggy algorithm implementations, including their correct versions and test cases.”
- A syntax-guided edit decoder for neural program repair (2021, background): “0, 297 bugs from IntroClassJava and 40 bugs from QuixBugs.”
- CoCoNuT: combining context-aware neural translation models using ensemble for program repair (2020, methodology): “- while ( true ) { + while (!queue. isEmpty()) { (b) Java patch for BREADTH_FIRST_SEARCH in QuixBugs . models is expensive, such models stay relevant for a time and do not need expensive retraining.”
- An Analysis of the Automatic Bug Fixing Performance of ChatGPT (2023, methodology): “To evaluate the automatic bug fixing performance of ChatGPT, we use the QuixBugs [12] benchmark set.”
- CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generation (2023, background): “Prenner and Robbes (2021) explore the CodeX’s (Chen et al., 2021) code repairing performance of Python and Java on QuixBugs (Lin et al., 2017).”