InCoder: A Generative Model for Code Infilling and Synthesis

By Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida I. Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke S. Zettlemoyer, Michael Lewis. 2022

909 citations recorded by the source corpus.

Code is seldom written in a single left-to-right pass and is instead repeatedly edited and refined. We introduce InCoder, a unified generative model that can perform program synthesis (via left-to-right generation) as well as editing (via infilling). InCoder is trained to generate code files from a large corpus of permissively licensed code, where regions of code have been randomly masked and moved to the end of each file, allowing code infilling with bidirectional context. Our model is the first generative model that is able to directly perform zero-shot code infilling, which we evaluate on challenging tasks such as type inference, comment generation, and variable re-naming. We find that the ability to condition on bidirectional context substantially improves performance on these tasks, while still performing comparably on standard program synthesis benchmarks in comparison to left-to-right only models pretrained at similar scale. The InCoder models and code are publicly released. https://sites.google.com/view/incoder-code-models

Subfields: Information Systems, Software, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

In Software

In Large Language Models

In Natural Language Processing

Citations in the corpus

LLaMA: Open and Efficient Foundation Language Models (2023, methodology): “Other models trained specifically for code also perform better than general models on these tasks (Chen et al., 2021; Nijkamp et al., 2022; Fried et al., 2022).”
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022, neutral): “Non-BLOOM results come from prior work (Chen et al., 2021; Fried et al., 2022).”
Code Llama: Open Foundation Models for Code (2023, methodology): “Our code-training for 7B, 13B and 70B Code Llama models features a multitask objective (Fried et al., 2023) consisting of both autoregressive and causal infilling prediction, enabling applications such as real-time completion in source code editors or docstring generation.”
Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling (2023, background): “…made big splashes as generative models in areas as diverse as text-to-image synthesis (Ramesh et al., 2022; Crowson et al., 2022; Rombach et al., 2022), protein modeling (Jumper et al., 2021; Ahdritz et al., 2022), and computer programming (Chen et al., 2021; Xu et al., 2022; Fried et al., 2022).”
Learning by Distilling Context (2022, background): “…4) shows that context distillation can be used to internalize Text-to-SQL training examples from the SPIDER dataset (Yu et al., 2018) into Incoder (Fried et al., 2022), and it outperforms directly learning with gradient descent by 9% for 8-shot adaptation; additionally, we show that as we distill…”
StarCoder: may the source be with you! (2023, methodology): “Decoder-only Transformer architectures have produced strong generative models of code, typically by training on mixtures of text and code from GitHub (Chen et al., 2021; Austin et al., 2021; Fried et al., 2022; Zheng et al., 2023; Nijkamp et al., 2023).”
MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation (2023, methodology): “Since 2020, several code generation models have been trained or fine-tuned on programming language text [1], [3], [4], [6], [16], [17].”
CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X (2023, methodology): “Second, in addition to code generation and code completion as Codex and others, CodeGeeX supports the tasks of code explanation and code translation between language pairs (Cf. Figure 1 (a)).”
Do Machine Learning Models Produce TypeScript Types that Type Check? (2023, methodology): “Although this paper does not evaluate these models, they are most closely related to InCoder [18], which is a general-purpose code generation model that we do evaluate.”
SantaCoder: don't reach for the stars! (2023, methodology): “Fill-in-the-middle evaluation To evaluate ﬁll-in-the-middle, we use the single-line exact match metric, which was introduced by Fried et al. (2022) and also employed by Bavarian et al. (2022).”
The Stack: 3 TB of permissively licensed source code (2022, methodology): “While Austin et al. (2021) include all three unit tests in the prompt, Fried et al. (2022) only include a single unit test because they observed that smaller language models (i.e., with a few billion parameters) did not benefit from more unit tests.”
DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation (2022, methodology): “We experiment with three families of pre-trained models: Codex, InCoder (Fried et al., 2022), and Code-Gen (Nijkamp et al., 2022).”
LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (2024, background): “These include CodeT5 (Wang et al., 2021, 2023), CodeGen (Nijkamp et al., 2022), SantaCoder (Allal et al., 2023), StarCoder (Li et al., 2023b), AlphaCode (Li et al., 2022), InCoder (Fried et al., 2022), and CodeGeeX (Zheng et al., 2023).”
Efficient Training of Language Models to Fill in the Middle (2022, methodology): “Notably, Donahue et al. [2020] explore infilling spans of varying granularities, such as words, sentences, or paragraphs, and InCoder [Fried et al., 2022] uses a similar evaluation framework to ours by studying infilling capabilities on sampling based benchmarks created from HumanEval [Chen et al.,…”
CodeT: Code Generation with Generated Tests (2022, methodology): “Other open-source code generation models include GPT-Neo (Black et al., 2021), GPT-J (Wang & Komatsuzaki, 2021), CodeParrot (Tunstall et al., 2022), PolyCoder (Xu et al., 2022), C ODE G EN (Nijkamp et al., 2022), and I N C ODER (Fried et al., 2022a).”
Multi-lingual Evaluation of Code Generation Models (2022, methodology): “In example 3, given result[char] = 1 in the right context, InCoder model generate result[char]+ = 1.”
CodeT5+: Open Code Large Language Models for Code Understanding and Generation (2023, methodology): “…[Chowdhery et al., 2022], GPT-4 [OpenAI, 2023], Codex [Chen et al., 2021], LLaMA [Touvron et al., 2023], CodeGen [Nijkamp et al., 2023b], Incoder [Fried et al., 2022], GPT-J [Wang and Komatsuzaki, 2021], GPT-Neo and GPT-NeoX [Black et al., 2022], MIM [Nguyen et al., 2023], CodeGeeX [Zheng et…”
Few-shot training LLMs for project-specific code-summarization (2022, methodology): “[9] introduce a large language model, InCoder, and try zero-shot training on CodeXGLUE Python dataset.”