Automatically generating commit messages from diffs using neural machine translation

By Siyuan Jiang, Ameer Armaly, Collin McMillan. 2017

292 citations recorded by the source corpus.

Commit messages are a valuable resource in comprehension of software evolution, since they provide a record of changes such as feature additions and bug repairs. Unfortunately, programmers often neglect to write good commit messages. Different techniques have been proposed to help programmers by automatically writing these messages. These techniques are effective at describing what changed, but are often verbose and lack context for understanding the rationale behind a change. In contrast, humans write messages that are short and summarize the high level rationale. In this paper, we adapt Neural Machine Translation (NMT) to automatically "translate" diffs into commit messages. We trained an NMT algorithm using a corpus of diffs and human-written commit messages from the top 1k Github projects. We designed a filter to help ensure that we only trained the algorithm on higher-quality commit messages. Our evaluation uncovered a pattern in which the messages we generate tend to be either very high or very low quality. Therefore, we created a quality-assurance filter to detect cases in which we are unable to produce good messages, and return a warning instead.

Subfields: Artificial Intelligence, Information Systems, Software

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Software

Citations in the corpus

A Novel Neural Source Code Representation Based on Abstract Syntax Tree (2019, methodology): “Many software engineering methods, such as source code classiﬁcation [1], [2], code clone detection [3], [4], [5], [6], defect prediction [7], [8] and code summarization [9], [10] have been proposed to improve software development and maintenance.”
Can we generate shellcodes via natural language? An empirical study (2021, background): “NMT is an emerging approach for code generation [88,56] and other programming tasks, such as code completion [23,77], the generation of UNIX commands [52,53] or commit messages [44,58,45], etc.”
Studying the Usage of Text-To-Text Transfer Transformer to Support Code-Related Tasks (2021, methodology): “The proposed summarization techniques fall into two categories: extractive [43]– [46] and abstractive [9], [11], [47]–[49].”
On Learning Meaningful Assert Statements for Unit Test Cases (2020, methodology): “We use the BLEU-4 variant, meaning that the BLEU score is computed by considering the 4-grams in the generated text, as previously done in other software-related tasks in the literature [11, 14].”
On Learning Meaningful Code Changes Via Neural Machine Translation (2019, background): “…on source code with the goal of (semi)automating several non-trivial tasks, including code completion [65], generation of commit messages [47], method names [27], code comments [66], defect prediction [63], bug localization [49] and ﬁxing [62], clone detection [64], code search [40], and…”
A Neural Model for Generating Natural Language Summaries of Program Subroutines (2019, background): “Jiang et al. [20] and Loyola et al. [29] generate descriptions of code changes (i.e. commit messages).”
Deep Code Comment Generation (2018, methodology): “BLEU score is a widely-used accuracy measure for NMT [22] and has been used in software tasks evaluation [12, 20].”
CoaCor: Code Annotation for Code Retrieval with Reinforcement Learning (2019, methodology): “Earlier works tackled the problem using template-based approaches [37, 49] and topic n-grams models [38] while the recent techniques [2, 19–22, 31] are mostly built upon deep neural networks.”
CodeTrans: Towards Cracking the Language of Silicone's Code Through Self-Supervised Deep Learning and High Performance Computing (2021, methodology): “We used the dataset 9 from Jiang et al. (2017) based on 1,000 Java repositories having the most Github stars.”