An Empirical Study on Learning Bug-Fixing Patches in the Wild via Neural Machine Translation

By Michele Tufano, Cody Watson, Gabriele Bavota, Massimiliano Di Penta, Martin White, Denys Poshyvanyk. 2018

450 citations recorded by the source corpus.

Millions of open source projects with numerous bug fixes are available in code repositories. This proliferation of software development histories can be leveraged to learn how to fix common programming bugs. To explore such a potential, we perform an empirical study to assess the feasibility of using Neural Machine Translation techniques for learning bug-fixing patches for real defects. First, we mine millions of bug-fixes from the change histories of projects hosted on GitHub in order to extract meaningful examples of such bug-fixes. Next, we abstract the buggy and corresponding fixed code, and use them to train an Encoder-Decoder model able to translate buggy code into its fixed version. In our empirical investigation, we found that such a model is able to fix thousands of unique buggy methods in the wild. Overall, this model is capable of predicting fixed patches generated by developers in 9--50% of the cases, depending on the number of candidate patches we allow it to generate. Also, the model is able to emulate a variety of different Abstract Syntax Tree operations and generate candidate patches in a split second.

Subfields: Information Systems, Software, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

In Software

In Natural Language Processing

Citations in the corpus

Evaluating Large Language Models Trained on Code (2021, background): “More recent works (Tufano et al., 2019; Drain et al., 2021) considered bug-ﬁxing as neural machine translation from buggy to correct programs.”
CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation (2021, methodology): “We employ two Java datasets provided by Tufano et al. (2019) with various function lengths: small (fewer than 50 tokens) and medium (50-100 tokens).”
Unified Pre-training for Program Understanding and Generation (2021, neutral): “, 2021) Java to C# 10,300 500 1,000 C# to Java 10,300 500 1,000 Program Repair Javasmall 46,680 5,835 5,835 (Tufano et al., 2019) Javamedium 52,364 6,545 6,545”
CodeBLEU: a Method for Automatic Evaluation of Code Synthesis (2020, positive): “We use the dataset released by Tufano et al. (2019).”
Unit Test Case Generation with Transformers (2020, background): “Deep learning techniques have shown the potential of learning from real-world examples, and have been employed in several software engineering tasks, such as code completion [17], automated patch generation [18], [19], comment generation [20], and many others [21].”
SequenceR: Sequence-to-Sequence Learning for End-to-End Program Repair (2018, methodology): “In order to compare against [10], we select those 1,116 tasks from CodRep4 where the buggy line resides in a method smaller than 100 tokens.”
DOBF: A Deobfuscation Pre-Training Objective for Programming Languages (2021, methodology): “Model Architecture We consider a seq2seq model with attention, composed of an encoder and a decoder using a transformer architecture Vaswani et al. [2017]. We train models with the same architecture and tokenizer as CodeBERT Feng et al.”
GraphCodeBERT: Pre-training Code Representations with Data Flow (2020, methodology): “We use the dataset released by Tufano et al. (2019) and report results in the Table 4.”
CoCoMIC: Code Completion by Jointly Modeling In-file and Cross-file Context (2022, unknown): “, 2022) to facilitate software engineering applications (Husain et al., 2019; Iyer et al., 2018; Tufano et al., 2019; Zhou et al., 2019).”
METHODS2TEST: A dataset of focal methods mapped to test cases (2022, methodology): “Examples of such datasets include Defects4J [14], Bugs2Fix [27], and CodRep [5] for automated bug-fixing, NASA Metrics Data Program (MDP) [23], PROMISE [2], and the unified dataset [9] for defect prediction, Landfill [19] for code smell detection, and many others.”
Leveraging Automated Unit Tests for Unsupervised Code Translation (2021, methodology): “In recent years, deep learning methods have been used to tackle various tasks in software engineering, with a particular interest in bug detection and repair (Wang et al., 2018; Chen et al., 2019; Allamanis et al., 2018; Tarlow et al., 2020; Murali et al., 2021; Dinella et al., 2020; Yasunaga & Lian”
To What Extent do Deep Learning-based Code Recommenders Generate Predictions by Cloning Code from the Training Set? (2022, methodology): “with what has been reported in the literature for applications of DL models to automate other code-related tasks, such as bug-fixing [56] or code review automation [57]; (ii) the code generation tasks on which we tested T5 included the generation of code blocks up to six lines of code, thus being no”
A Survey on Pretrained Language Models for Neural Code Intelligence (2022, background): “Code refinement (Tufano et al., 2019; Drain et al., 2021) aims to fix bugs in the given code snippet and output a bug-free one.”
InferFix: End-to-End Program Repair with LLMs (2023, background): “Patches in the Wild [32], utilize supervised machine translation to learn bug-fixing patterns for various common code defects.”
Examining Zero-Shot Vulnerability Repair with Large Language Models (2021, methodology): “NMT-based approaches train dedicated models, such as recurrent neural network (RNN)-based encoder-decoder architectures [15] on explicit “bug-fix pairs” mined from software repositories (e.”
xCodeEval: A Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrieval (2023, methodology): “The tuple ( C k , C ∗ j , P, V k ) represents a sample in our APR task.”