Big Code != Big Vocabulary: Open-Vocabulary Models for Source Code

By Rafael-Michael Karampatsis, Hlib Babii, Romain Robbes, Charles Sutton, Andrea Janes. 2020

235 citations recorded by the source corpus.

Statistical language modeling techniques have successfully been applied to large source code corpora, yielding a variety of new software development tools, such as tools for code suggestion, improving readability, and API migration. A major issue with these techniques is that code introduces new vocabulary at a far higher rate than natural language, as new identifier names proliferate. Both large vocabularies and out-of-vocabulary issues severely affect Neural Language Models (NLMs) of source code, degrading their performance and rendering them unable to scale. In this paper, we address this issue by: 1) studying how various modelling choices impact the resulting vocabulary on a large-scale corpus of 13,362 projects; 2) presenting an open vocabulary source code NLM that can scale to such a corpus, 100 times larger than in previous work; and 3) showing that such models outperform the state of the art on three distinct code corpora (Java, C, Python). To our knowledge, these are the largest NLMs for code that have been reported. All datasets, code, and trained models used in this work are publicly available.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Natural Language Processing

Citations in the corpus

Program Synthesis with Large Language Models (2021, background): “Code completion has been a particular focus of interest [Raychev et al., 2016, Karampatsis et al., 2020, Svyatkovskiy et al., 2020, Kim et al., 2020].”
A systematic evaluation of large language models of code (2022, background): “Recently, LMs have shown impressive performance in modeling also source code, written in programming languages (Hindle et al., 2016; Hellendoorn & Devanbu, 2017; Alon et al., 2020; Karampatsis et al., 2020).”
CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation (2021, methodology): “[42], using 1% of the subset in the corpus.”
Patching as Translation: the Data and the Metaphor (2020, methodology): “ng’]). In programming languages, vocabulary innovation is even more rampant, as developers tend to name a variable or method using a combination of words (e.g.,isNullOrEmpty) [12]. Karampatsis et al. [18] show that BPE can effectively address this issue in big code applications, so we apply this to ”
To What Extent do Deep Learning-based Code Recommenders Generate Predictions by Cloning Code from the Training Set? (2022, background): “The former is prone to the out-of-vocabulary problem [32] while the latter struggles with long”
Log-based Anomaly Detection Without Log Parsing (2021, background): “For example, space/stemming/camel case based tokenization strategies can lead to many OOV words and a big vocabulary [41].”
An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation (2023, methodology): “Neural techniques are rapidly being adopted for solving various Software Engineering problems, with promising results in several domains including code completion [34], [35], [36], [37], [38], program repair [39], [40], [41], and bug-ﬁnding [42], [43].”
Multi-task Learning based Pre-trained Language Model for Code Completion (2020, methodology): “The punctuation tokens are much easier to complete than identifiers, but these completions are not that useful for developers [25].”
Learning to Generate Code Sketches (2021, background): “Casalnuovo et al. [6], Karampatsis et al. [15] have observed a similar phenomenon on the perplexity across (standard left-to-right) language models for different programming languages.”
Code Prediction by Feeding Trees to Transformers (2020, methodology): “These representations have ranged from linear token sequence (as for code prediction [5], [7], [32]), to paths in an AST [15], [21], [52], and sometimes even ways to convey static analysis information to the neural network [24], [49], [51], [64].”
Learning to Predict User-Defined Types (2023, methodology): “Tokenizing the code, with Byte Pair Encoding (BPE) [28] or unigram language modeling [27] is common to manage large input code vocabularies [19].”
LineVul: A Transformer-based Line-Level Vulnerability Prediction (2022, background): “[27] conduct a study of how different modelling choices affects the model performance of language models of source code and find the advancement of leveraging BPE model for language modeling of source code.”
CodeFill: Multi-token Code Completion by Jointly learning from Structure and Naming Sequences (2022, methodology): “[24] presented a large-scale open-vocabulary neural LM.”