Learning Python Code Suggestion with a Sparse Pointer Network

By Avishkar Bhoopchand, Tim Rocktaschel, Earl T. Barr, Sebastian Riedel. 2016

90 citations recorded by the source corpus.

To enhance developer productivity, all modern integrated development environments (IDEs) include code suggestion functionality that proposes likely next tokens at the cursor. While current IDEs work well for statically-typed languages, their reliance on type annotations means that they do not provide the same level of support for dynamic programming languages as for statically-typed languages. Moreover, suggestion engines in modern IDEs do not propose expressions or multi-statement idiomatic code. Recent work has shown that language models can improve code suggestion systems by learning from software repositories. This paper introduces a neural language model with a sparse pointer network aimed at capturing very long range dependencies. We release a large-scale code suggestion corpus of 41M lines of Python code crawled from GitHub. On this corpus, we found standard neural language models to perform well at suggesting local phenomena, but struggle to refer to identifiers that are introduced many tokens in the past. By augmenting a neural language model with a pointer network specialized in referring to predefined classes of identifiers, we obtain a much lower perplexity and a 5 percentage points increase in accuracy for code suggestion compared to an LSTM baseline. In fact, this increase in code suggestion accuracy is due to a 13 times more accurate prediction of identifiers. Furthermore, a qualitative analysis shows this model indeed captures interesting long-range dependencies, like referring to a class member defined over 60 tokens in the past.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Natural Language Processing

Citations in the corpus

A Survey of Machine Learning for Big Code and Naturalness (2017, methodology): “[26] PD (c) Token NN (Pointer Net) Code Completion”
Unsupervised Translation of Programming Languages (2020, background): “Another line of work studied the applications of neural networks to code suggestion [2, 11, 34], or error detection [13, 18, 47].”
A Parallel Corpus of Python Functions and Documentation Strings for Automated Code Documentation and Code Generation (2017, background): “…baseline systems (Oda et al., 2015; Yin and Neubig, 2017), with BLEU scores more than twice those of machine translation between natural languages (Cettolo et al., 2016), a task that we would expect to be no more difficult than code documentation or code generation, especially given the much…”
Learning to Represent Programs with Graphs (2017, methodology): “For example, Hindle et al. (2012); Bhoopchand et al. (2016) model the code as a sequence of tokens, while Maddison & Tarlow (2014); Raychev et al. (2016) model the syntax tree structure of code.”
Neural Program Synthesis with Priority Queue Training (2018, background): “…2015; Kaiser & Sutskever, 2016; Kurach et al., 2015; Neelakantan et al., 2016; Reed & de Freitas, 2016; Andreas et al., 2016; Balog et al., 2017; Bhoopchand et al., 2016; Gaunt et al., 2016; Khanh Dam et al., 2016; Neelakantan et al., 2017; Riedel et al., 2016; Schaechtle et al., 2016; Zaremba…”
Learning to Fix Build Errors with Graph2Diff Neural Networks (2019, background): “Graph2Tocopo is a single abstraction that unifies ideas from many recent works in modelling source code [3, 4, 8, 22, 35, 44].”
Semantic Code Repair using Neuro-Symbolic Transformation Networks (2017, methodology): “Our model can also be thought of as an evolution of work on neural code completion and summarization [2, 7].”
Language-Agnostic Representation Learning of Source Code from Structure and Context (2021, methodology): “They can be used as advanced auto-completion tools (Hindle et al., 2012; Bhoopchand et al., 2016), including for user-provided tokens like Variable Names (Raychev et al., 2014; Allamanis et al., 2014).”
Multi-task Learning based Pre-trained Language Model for Code Completion (2020, methodology): “to source code modeling [3, 27].”
SmartPaste: Learning to Adapt Source Code (2017, unknown): “Bhoopchand et al. [6], Hindle et al. [10] model the code as a sequence of tokens, while Maddison and Tarlow [11], Raychev et al. [17] model the syntax tree structure of code.”
Code Completion with Neural Attention and Pointer Networks (2017, background): “Torender effective code completion for dynamically-typed languages, recently, researchers turn to learning-based language models (Hindle et al. 2012; White et al. 2015; Bielik, Raychev, and Vechev 2016).”
Neural Networks for Modeling Source Code Edits (2018, methodology): “…code models based on n-grams (Hindle et al., 2012; Allamanis & Sutton, 2013), grammars (Allamanis & Sutton, 2014), neural networks (Raychev et al., 2014; White et al., 2015; Ling et al., 2016; Bhoopchand et al., 2016), and combinations of the two (Maddison & Tarlow, 2014; Yin & Neubig, 2017).”