Break-It-Fix-It: Unsupervised Learning for Program Repair

By Michihiro Yasunaga, Percy Liang. 2021

133 citations recorded by the source corpus.

We consider repair tasks: given a critic (e.g., compiler) that assesses the quality of an input, the goal is to train a fixer that converts a bad example (e.g., code with syntax errors) into a good one (e.g., code with no syntax errors). Existing works create training data consisting of (bad, good) pairs by corrupting good examples using heuristics (e.g., dropping tokens). However, fixers trained on this synthetically-generated data do not extrapolate well to the real distribution of bad inputs. To bridge this gap, we propose a new training approach, Break-It-Fix-It (BIFI), which has two key ideas: (i) we use the critic to check a fixer's output on real bad inputs and add good (fixed) outputs to the training data, and (ii) we train a breaker to generate realistic bad code from good code. Based on these ideas, we iteratively update the breaker and the fixer while using them in conjunction to generate more paired data. We evaluate BIFI on two code repair datasets: GitHub-Python, a new dataset we introduce where the goal is to repair Python code with AST parse errors; and DeepFix, where the goal is to repair C code with compiler errors. BIFI outperforms existing methods, obtaining 90.5% repair accuracy on GitHub-Python (+28.5%) and 71.7% on DeepFix (+5.6%). Notably, BIFI does not require any labeled data; we hope it will be a strong starting point for unsupervised learning of various repair tasks.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

InCoder: A Generative Model for Code Infilling and Synthesis (2022, methodology): “This includes methods to infer variable types [50, 66], generate unit tests [25], repair programs [30, 70, 20, 71], and verify program correctness [55].”
Code Llama: Open Foundation Models for Code (2023, methodology): “These tasks include code summarization, refinement, translation (Rozière et al., 2020; 2021; Szafraniec et al., 2023) fixing bugs (Yasunaga & Liang, 2021; Zhang et al., 2022a; Prenner et al., 2022), fixing build errors (Tarlow et al., 2020) or generating unit tests (Tufano et al., 2020; Li et al.,…”
Learning to repair: Repairing model output errors after deployment using a dynamic memory of feedback (2021, unknown): “These methods all augment/replace the standard use of automated answer feedback (if available), e.g., testing whether a semantic parse correctly executes to the correct answer, e.g., (Zettlemoyer and Collins, 2005), sometimes using unsupervised techniques to generate additional training data, e.g., ”
Teaching Large Language Models to Self-Debug (2023, methodology): “Program repair is an area of research concerned with fixing bugs in code, where several neural network models have been developed to handle different types of bugs [21, 60, 20, 53].”
Large Language Models as Analogical Reasoners (2023, methodology): “We scraped data from codeforces.com, following the procedure in prior works (Li et al., 2022b; Kulal et al., 2019; Yasunaga & Liang, 2021).”
Repairing Bugs in Python Assignments Using Large Language Models (2022, methodology): “In particular, for programming education past research has explored topics including feedback generation [9]– [12], [29], [41], [42] and program repair [20], [36], [43]–[46].”
Measuring The Impact Of Programming Language Distribution (2023, unknown): “Recent works have introduced new execution-based benchmarks for both generation (Austin et al., 2021; Hendrycks et al., 2021; Chen et al., 2021; Lai et al., 2022) and repair (Yasunaga & Liang, 2021) tasks, however, these have been limited to only Python.”
Demystifying GPT Self-Repair for Code Generation (2023, methodology): “Statistical and learning-based techniques for code repair have a rich history in both the programming languages and machine learning communities, although they have traditionally been used predominantly to repair human-written code [3, 8, 19, 22, 24, 33, 35].”
Baldur: Whole-Proof Generation and Repair with Large Language Models (2023, background): “It is possible to learn how to repair errors together by learning how to create errors, which can increase the amount of available training data, but poses an additional challenge of learning to approximate making human-like errors [97].”