A Convolutional Attention Network for Extreme Summarization of Source Code

By Miltiadis Allamanis, Hao Peng, Charles Sutton. 2016

626 citations recorded by the source corpus.

Attention mechanisms in neural networks have proved useful for problems in which the input and output do not have fixed dimension. Often there exist features that are locally translation invariant and would be valuable for directing the model's attention, but previous attentional architectures are not constructed to learn such features specifically. We introduce an attentional neural network that employs convolution on the input tokens to detect local time-invariant and long-range topical attention features in a context-dependent way. We apply this architecture to the problem of extreme summarization of source code snippets into short, descriptive function name-like summaries. Using those features, the model sequentially generates a summary by marginalizing over two attention mechanisms: one that predicts the next summary token based on the attention weights of the input tokens and another that is able to copy a code token as-is directly into the summary. We demonstrate our convolutional attention neural network's performance on 10 popular Java projects showing that it achieves better performance compared to previous attentional mechanisms.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Natural Language Processing

In Signal Processing

Citations in the corpus

Program Synthesis with Large Language Models (2021, background): “, 2019], predicting readable names to describe functions [Allamanis et al., 2016], and generating function text from docstrings [Iyer et al.”
The adverse effects of code duplication in machine learning models of code (2018, background): “Java-Small [5], [3] 79.”
Unified Pre-training for Program Understanding and Generation (2021, background): “Code Summarization (MovshovitzAttias and Cohen, 2013; Allamanis et al., 2016; Iyer et al., 2016; Alon et al., 2019a; Hu et al., 2018; Harer et al., 2019; Ahmad et al., 2020), Bug Detection (Ray et al., 2016; Li et al., 2018b; Russell et al., 2018; Zhou et al., 2019; Chakraborty et al., 2020),…”
CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation (2021, background): “• code-code (clone detection [65, 84, 46, 80, 9, 89, 86], defect detection [91, 55, 51, 42, 78, 79], cloze test [15], code completion [57, 1, 56, 66, 67, 8, 27, 7], code repair [69, 4, 23, 26, 71, 73], and code-to-code translation [48, 37, 10, 41]) • text-code (natural language code search [32, 20, ”
CodeSearchNet Challenge: Evaluating the State of Semantic Code Search (2019, neutral): “[3], Alon et al.”
code2seq: Generating Sequences from Structured Representations of Code (2018, methodology): “As was previously observed (Allamanis et al., 2016; Alon et al., 2018a), this is a good benchmark because a method name in open-source Java projects tends to be succinct and precise, and a method body is often a complete logical unit.”
Learning to Represent Programs with Graphs (2017, background): “…deep learning models of source code capture its shallow, textual structure, e.g. as a sequence of tokens (Hindle et al., 2012; Raychev et al., 2014; Allamanis et al., 2016), as parse trees (Maddison & Tarlow, 2014; Bielik et al., 2016), or as a ﬂat dependency networks of variables (Raychev et al.,…”
Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflow (2018, background): “Recent years have witnessed the burgeoning of a new suite of developer assistance tools based on natural language processing (NLP) techniques, for code completion [9], source code summarization [2], automatic documentation of source code [44], deobfuscation [16, 34, 40], cross-language porting [27, ”
code2vec: learning distributed representations of code (2018, methodology): “…[Allamanis et al. 2015a; Alon et al. 2018; Raychev et al. 2015], code completion [Mishne et al. 2012; Raychev et al. 2014], code summarization [Allamanis et al. 2016], code generation [Amodio et al. 2017; Lu et al. 2017; Maddison and Tarlow 2014; Murali et al. 2017], and more (see [Allamanis…”
Reference-Aware Language Models (2016, methodology): “systems while our model queries the database directly. Previous work on recipe generation from ingredients was proposed in (Kiddon et al., 2016). This model extents previous work on attention models (Allamanis et al., 2016) to checklists, where as our proposed model with explicit references to those”
DOBF: A Deobfuscation Pre-Training Objective for Programming Languages (2021, methodology): “Following previous works (Allamanis et al., 2015; 2016; Alon et al., 2018; 2019), we also report the subtoken score, a more flexible metric which computes the precision, recall, and F1 scores for retrieving the original case-insensitive subtokens.”
Explainable Prediction of Medical Codes from Clinical Text (2018, methodology): “Our attention mechanism is most similar to those of Yang et al. (2016) and Allamanis et al. (2016), in that we use context vectors to compute attention over spe-ciﬁc locations in the text.”
DeepBugs: a learning approach to name-based bug detection (2018, methodology): “Another approach summarizes code into descriptive names that can serve, e.g., as method names [Allamanis et al. 2016].”
JuICe: A Large Scale Distantly Supervised Dataset for Open Domain Context-based Code Generation (2019, background): “Although various large scale datasets (Allamanis and Sutton, 2013, 2014; Allamanis et al., 2016; Iyer et al., 2018) to study code generation have been cre-”
Big Code != Big Vocabulary: Open-Vocabulary Models for Source Code (2020, methodology): “also used for program repair [20], and method naming [5].”
Learning Continuous Semantic Representations of Symbolic Expressions (2016, background): “Allamanis et al. (2016) learn variable-sized representations of source code snippets to summarize them with a short function-like name but aim learn summarization features in code rather than representations of symbolic expression equivalence.”
Learning to Fix Build Errors with Graph2Diff Neural Networks (2019, background): “Graph2Tocopo is a single abstraction that unifies ideas from many recent works in modelling source code [3, 4, 8, 22, 35, 44].”
Retrieval-based Neural Source Code Summarization (2020, methodology): “The convolutional attention model is presented to predict extreme summarization of source code such as function names [2].”