CodeFill: Multi-token Code Completion by Jointly learning from Structure and Naming Sequences

By Maliheh Izadi, Roberta Gismondi, Georgios Gousios. 2022

128 citations recorded by the source corpus.

Code completion is an essential feature of IDEs, yet current auto-completers are restricted to either grammar-based or NLP-based single token completions. Both approaches have significant draw-backs: grammar-based autocompletion is restricted in dynamically-typed language environments, whereas NLP-based autocompleters struggle to understand the semantics of the programming language and the developer's code context. In this work, we present CodeFill, a language model for autocompletion that combines learned structure and naming information. Using a parallel Transformer architecture and multi-task learning, CodeFill consumes sequences of source code token names and their equivalent AST token types. Uniquely, CodeFill is trained both for single-token and multi-token (statement) prediction, which enables it to learn long-range dependencies among grammatical and naming elements. We train CodeFill on two datasets, consisting of 29M and 425M lines of code, respectively. To make the evaluation more realistic, we develop a method to automatically infer points in the source code at which completion matters. We compare CodeFill against four baselines and two state-of-the-art models, GPT-C and TravTrans+. CodeFill surpasses all baselines in single token prediction (MRR: 70.9% vs. 66.2% and 67.8%) and outperforms the state of the art for multi-token prediction (ROUGE-L: 63.7% vs. 52.4% and 59.2%, for $n=4$ tokens). We publicly release our source code and datasets.

Subfields: Artificial Intelligence, Information Systems, Code Generation & Program Synthesis, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Code Generation & Program Synthesis

In Signal Processing

Citations in the corpus

InCoder: A Generative Model for Code Infilling and Synthesis (2022, neutral): “, decoder-only language models [10, 18, 35, 68, 46], encoder-only masked language models [24, 37], and encoder-decoder models [4, 44, 54, 64].”
SantaCoder: don't reach for the stars! (2023, background): “…models with a causal language modeling objective (Chen et al., 2021; Austin et al., 2021; Nijkamp et al., 2022; Christopoulou et al., 2022; Izadi et al., 2022; Xu et al., 2022b; Athiwaratkun et al., 2022), while other studies have investigated encoder (Feng et al., 2020a; Kanade et al.,…”
The Stack: 3 TB of permissively licensed source code (2022, background): “Several groups have explored decoder-only models with a causal language modeling objective (Chen et al., 2021; Austin et al., 2021; Nijkamp et al., 2022; Christopoulou et al., 2022; Izadi et al., 2022; Xu et al., 2022) and generally found that larger models are increasingly capable of synthesizing p”
Large Language Models for Software Engineering: A Systematic Literature Review (2023, methodology): “A notable trend was the customization of LLMs for precise SE tasks [107, 150, 328].”
Late Code Chunking: A Code Chunking Strategy for Repository-Level Code Completion (2026, unknown): “The content of a given chunk is fundamentally determined by its preceding context (Izadi et al., 2022); consequently, a high correlation exists be-tween the content that immediately precedes a segment and the segment itself.”