Code Completion with Neural Attention and Pointer Networks

By Jian Li, Yue Wang, Michael R. Lyu, Irwin King. 2017

246 citations recorded by the source corpus.

Intelligent code completion has become an essential research task to accelerate modern software development. To facilitate effective code completion for dynamically-typed programming languages, we apply neural language models by learning from large codebases, and develop a tailored attention mechanism for code completion. However, standard neural language models even with attention mechanism cannot correctly predict the out-of-vocabulary (OoV) words that restrict the code completion performance. In this paper, inspired by the prevalence of locally repeated terms in program source code, and the recently proposed pointer copy mechanism, we propose a pointer mixture network for better predicting OoV words in code completion. Based on the context, the pointer mixture network learns to either generate a within-vocabulary word through an RNN component, or regenerate an OoV word from local context through a pointer component. Experiments on two benchmarked datasets demonstrate the effectiveness of our attention mechanism and pointer mixture network on the code completion task.

Subfields: Computer Networks and Communications, Information Systems, Natural Language Processing, Code Generation & Program Synthesis, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Information Systems

In Natural Language Processing

In Code Generation & Program Synthesis

In Signal Processing

Citations in the corpus

You Autocomplete Me: Poisoning Vulnerabilities in Neural Code Completion (2020, background): “Code completion systems based on language models that generate code tokens [3, 34, 49, 57], rather than natural-language tokens, are the basis of intelligent IDEs [11] such as Deep TabNine [16] and Microsoft’s Visual Studio IntelliCode [45].”
Unified Pre-training for Program Understanding and Generation (2021, background): “…et al., 2016; Iyer et al., 2016; Alon et al., 2019a; Hu et al., 2018; Harer et al., 2019; Ahmad et al., 2020), Bug Detection (Ray et al., 2016; Li et al., 2018b; Russell et al., 2018; Zhou et al., 2019; Chakraborty et al., 2020), Program Repair (Chen et al., 2019; Chakraborty et al., 2020;…”
Unsupervised Translation of Programming Languages (2020, background): “Another line of work studied the applications of neural networks to code suggestion [2, 11, 34], or error detection [13, 18, 47].”
DOBF: A Deobfuscation Pre-Training Objective for Programming Languages (2021, methodology): “Machine learning is frequently used on tasks involving programming languages, including code completion [28, 33, 35, 42], bug detection and code repair [15, 38, 46], code summarization [8, 25], clone detection [2, 47, 48], code search [13, 23] and code translation [14, 41].”
GraphCodeBERT: Pre-training Code Representations with Data Flow (2020, background): “Neural Networks with Code Structure In recent years, some neural networks leveraging code structure such as AST have been proposed and achieved strong performance in code-related tasks like code completion (Li et al., 2017; Alon et al., 2019; Kim et al., 2020), code generation (Rabinovich et al.”
Big Code != Big Vocabulary: Open-Vocabulary Models for Source Code (2020, background): “RNNs with gated units can learn when to forget information from the hidden state and take newer, more important information into account [45].”
Leveraging Automated Unit Tests for Unsupervised Code Translation (2021, methodology): “…years, deep learning methods have been used to tackle various tasks in software engineering, with a particular interest in bug detection and repair (Wang et al., 2018; Chen et al., 2019; Allamanis et al., 2018; Tarlow et al., 2020; Murali et al., 2021; Dinella et al., 2020; Yasunaga & Liang, 2020;…”
SpreadsheetCoder: Formula Prediction from Semi-structured Context (2021, methodology): “Our spreadsheet formula prediction problem is also related to code completion tasks (Raychev et al., 2014; Li et al., 2018; Svyatkovskiy et al., 2019; 2020; Svyatkovskoy et al., 2020).”
To What Extent do Deep Learning-based Code Recommenders Generate Predictions by Cloning Code from the Training Set? (2022, methodology): “[39] proposed a pointer mixture deep learning model for Python that leverages the pointer copy mechanism to address the out-of-vocabulary problem.”
MISIM: An End-to-End Neural Code Similarity System (2020, methodology): “pace of code similarity. A large body of work has studied applying machine learning to learn from the AST for completing various tasks on code (Alon et al., 2019a; Chen et al., 2018; Hu et al., 2018; Li et al., 2018; Mou et al., 2016). Odena & Sutton (2020) represent a program as property signat”
An Empirical Evaluation of GitHub Copilot's Code Suggestions (2022, background): “While there is a lot of work on code synthesis and code search using deep learning (e.g., [3–5, 8, 12, 14, 19, 32]), we do not dive into the specifics here since we focus on evaluating Copilot’s capabilities while treating it as a black box.”
CodeFill: Multi-token Code Completion by Jointly learning from Structure and Naming Sequences (2022, methodology): “[29], propose a pointer mixture model to address the OOV problem.”
Code Prediction by Feeding Trees to Transformers (2020, methodology): “The most common neural technique for code prediction is Recurrent Neural Networks (RNNs) [4] and their variants [5]–[10], where the code represented as a linear sequence is fed as input to the model.”
PlotCoder: Hierarchical Decoding for Synthesizing Visualization Code in Programmatic Context (2021, methodology): “, autocompleting the program given the code context (Raychev et al., 2014; Li et al., 2018; Svyatkovskiy et al., 2020).”
Maybe Deep Neural Networks are the Best Choice for Modeling Source Code (2019, background): “Lastly, [48] attempt to improve code completion performance on OOV words by augmenting an RNN with a pointer network [72].”
An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation (2023, methodology): “Neural techniques are rapidly being adopted for solving various Software Engineering problems, with promising results in several domains including code completion [34], [35], [36], [37], [38], program repair [39], [40], [41], and bug-ﬁnding [42], [43].”
Language-Agnostic Representation Learning of Source Code from Structure and Context (2021, unknown): “Other approaches (Alon et al., 2018; Li et al., 2017) also leverage the AST structure, but linearize the graph by ﬁrst traversing it.”
HELoC: Hierarchical Contrastive Learning of Source Code Representation (2022, methodology): “Token-Based Models: RtvNN [45] uses an RNN to learn the representations of code tokens and trains a recursive autoencoder to learn AST representations.”
Multi-task Learning based Pre-trained Language Model for Code Completion (2020, methodology): “ited by machine learning algorithms. Most of these models are based on learned representations called embeddings, which transform words into a continuous vector space [37]. However, existing research [3, 25, 27] uses static embeddings, which map a word to the same vector regardless of its context. F”