Modeling Vocabulary for Big Code Machine Learning

By Hlib Babii, Andrea Janes, Romain Robbes. 2019

21 citations recorded by the source corpus.

When building machine learning models that operate on source code, several decisions have to be made to model source-code vocabulary. These decisions can have a large impact: some can lead to not being able to train models at all, others significantly affect performance, particularly for Neural Language Models. Yet, these decisions are not often fully described. This paper lists important modeling choices for source code vocabulary, and explores their impact on the resulting vocabulary on a large-scale corpus of 14,436 projects. We show that a subset of decisions have decisive characteristics, allowing to train accurate Neural Language Models quickly on a large corpus of 10,106 projects.

Subfields: Artificial Intelligence, Information Systems, Code Generation & Program Synthesis

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Code Generation & Program Synthesis

Citations in the corpus

TypeWriter: neural type prediction with search-based validation (2019, methodology): “e is trained on sequences of tokens extracted from source code files, whileEword is trained on sequences of words extracted from comments. To mitigate the problem of large vocabularies in source code [5], TypeWriter preprocesses each identifier using a helper function norm(), which tokenizes, lemmat”
Big Code != Big Vocabulary: Open-Vocabulary Models for Source Code (2020, background): “ is a consolidation of two unpublished works originally conducted independently: one work focused on the impact of various vocabulary choices on the resulting vocabulary size and the training of NLMs [9], while the other work investigated the speciﬁc vocabulary choice of Byte-Pair Encoding, and intr”