Maybe Deep Neural Networks are the Best Choice for Modeling Source Code

By Rafael-Michael Karampatsis, Charles Sutton. 2019

56 citations recorded by the source corpus.

Statistical language modeling techniques have successfully been applied to source code, yielding a variety of new software development tools, such as tools for code suggestion and improving readability. A major issue with these techniques is that code introduces new vocabulary at a far higher rate than natural language, as new identifier names proliferate. But traditional language models limit the vocabulary to a fixed set of common words. For code, this strong assumption has been shown to have a significant negative effect on predictive performance. But the open vocabulary version of the neural network language models for code have not been introduced in the literature. We present a new open-vocabulary neural language model for code that is not limited to a fixed vocabulary of identifier names. We employ a segmentation into subword units, subsequences of tokens chosen based on a compression criterion, following previous work in machine translation. Our network achieves best in class performance, outperforming even the state-of-the-art methods of Hellendoorn and Devanbu that are designed specifically to model code. Furthermore, we present a simple method for dynamically adapting the model to a new test project, resulting in increased performance. We showcase our methodology on code corpora in three different languages of over a billion tokens each, hundreds of times larger than in previous work. To our knowledge, this is the largest neural language model for code that has been reported.

Subfields: Artificial Intelligence, Information Systems, Model Architectures, Code Generation & Program Synthesis

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Model Architectures

In Code Generation & Program Synthesis

Citations in the corpus

TypeWriter: neural type prediction with search-based validation (2019, background): “Very recent work for by Karampatsis and Sutton [16] uses sub-word information to account for neologisms with very good results.”
SequenceR: Sequence-to-Sequence Learning for End-to-End Program Repair (2018, methodology): “Another approach to addressing the vocabulary size problem in code is to use byte pair encoding (BPE), which has been widely used in NLP and also applied to source code [14].”
Big Code != Big Vocabulary: Open-Vocabulary Models for Source Code (2020, methodology): “ting vocabulary size and the training of NLMs [9], while the other work investigated the speciﬁc vocabulary choice of Byte-Pair Encoding, and introduced several improvements to the training procedure [55]. This paper contains joint work that improves on bothearlier worksbyinvestigatingadditionalchar”
Learning to Fix Build Errors with Graph2Diff Neural Networks (2019, background): “ficult problem [21]. Using pointers, the machine learning component can predict both where and how to fix. Finally, thecopy mechanism addresses the well-known out-of-vocabulary problem of source code [5, 16, 17]: source code projects often include projectspecific identifiers that do not occur in the”
Modeling Vocabulary for Big Code Machine Learning (2019, methodology): “In parallel to this work, Karampatsis and Sutton [40] also investigate the problem of large vocabularies in source code.”
Typilus: neural type hints (2020, background): “Code completion [32, 34, 36, 52] is one of the most widely explored topics.”
Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models (2022, background): “Karampatsis and Sutton [20] similarly introduced open-vocabulary models that can generate code with an arbitrary number of tokens.”
To What Extent do Deep Learning-based Code Recommenders Generate Predictions by Cloning Code from the Training Set? (2022, methodology): “This has been achieved, for example, by (i) exploiting contextual information such as the code surrounding the statements on which code completion is triggered [8, 48] and/or or the recent code change history [48]; and (ii) moving towards more advanced techniques such as Deep Learning (DL) models, t”