Graph-based, Self-Supervised Program Repair from Diagnostic Feedback

By Michihiro Yasunaga, Percy Liang. 2020

204 citations recorded by the source corpus.

We consider the problem of learning to repair programs from diagnostic feedback (e.g., compiler error messages). Program repair is challenging for two reasons: First, it requires reasoning and tracking symbols across source code and diagnostic feedback. Second, labeled datasets available for program repair are relatively small. In this work, we propose novel solutions to these two challenges. First, we introduce a program-feedback graph, which connects symbols relevant to program repair in source code and diagnostic feedback, and then apply a graph neural network on top to model the reasoning process. Second, we present a self-supervised learning paradigm for program repair that leverages unlabeled programs available online to create a large amount of extra program repair examples, which we use to pre-train our models. We evaluate our proposed approach on two applications: correcting introductory programming assignments (DeepFix dataset) and correcting the outputs of program synthesis (SPoC dataset). Our final system, DrRepair, significantly outperforms prior work, achieving 68.2% full repair rate on DeepFix (+22.9% over the prior best), and 48.4% synthesis success rate on SPoC (+3.7% over the prior best).

Subfields: Computer Networks and Communications, Information Systems, Software, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Information Systems

In Software

In Natural Language Processing

Citations in the corpus

PaLM: Scaling Language Modeling with Pathways (2022, methodology): “We observe this behavior qualitatively in the predictions, where PaLM-Coder is more likely than Codex to 13Prior approaches to DeepFix change at most 5 lines of code (Yasunaga & Liang, 2020, 2021).”
Program Synthesis with Large Language Models (2021, methodology): “Many learning methods have been proposed for program repair [Allamanis et al., 2018b, Tarlow et al., 2019, Hellendoorn et al., 2019, Dinella et al., 2019, Yasunaga and Liang, 2020, Chen et al., 2019b, Pradel and Sen, 2018].”
Break-It-Fix-It: Unsupervised Learning for Program Repair (2021, methodology): “Note that f0 here corresponds to the fixer learned in prior works (e.g., Gupta et al. (2017); Yasunaga & Liang (2020)).”
InCoder: A Generative Model for Code Infilling and Synthesis (2022, methodology): “This includes methods to infer variable types [50, 66], generate unit tests [25], repair programs [30, 70, 20, 71], and verify program correctness [55].”
Self-Refine: Iterative Refinement with Self-Feedback (2023, methodology): “, 2021), program synthesis (Le et al., 2022a; Yasunaga and Liang, 2020), and other tasks (Bai et al.”
Scaling Instruction-Finetuned Language Models (2022, background): “, 2022) and program synthesis data (Yasunaga and Liang, 2020; Li et al., 2022).”
UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models (2022, methodology): “Structured knowledge encoding methods have been explored in many prior studies (Bogin et al., 2019; Lin et al., 2019; Yin et al., 2020a; Herzig et al., 2020; Agarwal et al., 2020; Saxena et al., 2020; Yasunaga and Liang, 2020; Yasunaga et al., 2021; Oguz et al., 2021).”
WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020, neutral): “…various ways: automatic code completion (Raychev et al., 2014; Svyatkovskiy et al., 2019), program synthesis (Bunel et al., 2018; Kulal et al., 2019), program repair (Vasic et al., 2019; Yasunaga and Liang, 2020), code search (Husain et al., 2019), and code summarization (Allamanis et al., 2015).”
Leveraging Automated Unit Tests for Unsupervised Code Translation (2021, background): “…detection and repair (Wang et al., 2018; Chen et al., 2019; Allamanis et al., 2018; Tarlow et al., 2020; Murali et al., 2021; Dinella et al., 2020; Yasunaga & Liang, 2020; Tufano et al., 2019; Drain et al., 2021) and code completion (Li et al., 2018; Liu et al., 2020; Kim et al., 2021;…”
PLUR: A Unifying, Graph-Based View of Program Learning, Understanding, and Repair (2021, background): “…of graphs, copy mechanisms, pointers, and other custom architectures [Maddison and Tarlow, 2014, Bielik et al., 2016, Allamanis et al., 2016, Mukherjee et al., 2017, Yin and Neubig, 2017, Allamanis et al., 2018, Alon et al., 2018, Dinella et al., 2020, Yasunaga and Liang, 2020, Nye et al., 2020].”
A syntax-guided edit decoder for neural program repair (2021, methodology): “Several existing DL-based approaches also use the idea of generating edits on programs [4, 5, 9, 65, 74].”
Large Language Models as Analogical Reasoners (2023, methodology): “…et al., 2019; Feng et al., 2020; Zhang et al., 2022a; Yasunaga et al., 2021; 2022a; Xie et al., 2022), and incorporating external modules for reasoning such as logic and program execution (Chen et al., 2018; 2019; Yasunaga & Liang, 2020; Gupta et al., 2020; Ren et al., 2021; Zhang et al., 2023).”
Demystifying GPT Self-Repair for Code Generation (2023, methodology): “More recently, using repair as a post-processing step to improve code which was itself automatically synthesised has been used in the synthesis of both domain-specific languages [13] and general-purpose code [18, 34, 35].”
CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing (2023, unknown): “[25] Michihiro Yasunaga and Percy Liang.”
Learning to repair: Repairing model output errors after deployment using a dynamic memory of feedback (2021, unknown): “As the domain of the task becomes more specialized, such as database query generation (Elgohary et al., 2021) or code correction (Yasunaga and Liang, 2020), collecting data to train G becomes difﬁcult.”