Efficient Training of Language Models to Fill in the Middle

By Mo Bavarian, Heewoo Jun, Nikolas A. Tezak, John Schulman, Christine McLeavey, Jerry Tworek, Mark Chen. 2022

317 citations recorded by the source corpus.

We show that autoregressive language models can learn to infill text after we apply a straightforward transformation to the dataset, which simply moves a span of text from the middle of a document to its end. While this data augmentation has garnered much interest in recent years, we provide extensive evidence that training models with a large fraction of data transformed in this way does not harm the original left-to-right generative capability, as measured by perplexity and sampling evaluations across a wide range of scales. Given the usefulness, simplicity, and efficiency of training models to fill-in-the-middle (FIM), we suggest that future autoregressive language models be trained with FIM by default. To this end, we run a series of ablations on key hyperparameters, such as the data transformation frequency, the structure of the transformation, and the method of selecting the infill span. We use these ablations to prescribe strong default settings and best practices to train FIM models. We have released our best infilling model trained with best practices in our API, and release our infilling benchmarks to aid future research.

Subfields: Artificial Intelligence, General Social Sciences, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In General Social Sciences

In Large Language Models

In Natural Language Processing

Citations in the corpus

InCoder: A Generative Model for Code Infilling and Synthesis (2022, methodology): “In addition, concurrent to our work, OpenAI developed a fill-in-the-middle (FIM) training objective similar to the causal masking objective we use, trained code models with it, and evaluated on the HumanEval infilling tasks we introduce here (Bavarian et al., 2022).”
Large Language Models Are Human-Level Prompt Engineers (2022, methodology): “, 2022), and InsertGPT (Bavarian et al., 2022)—to infer the missing instructions.”
Code Llama: Open Foundation Models for Code (2023, methodology): “We observe a further increase in performance when prompting the models in SPM format, like witnessed in Bavarian et al. (2022).”
SantaCoder: don't reach for the stars! (2023, methodology): “Fill-in-the-middle evaluation To evaluate ﬁll-in-the-middle, we use the single-line exact match metric, which was introduced by Fried et al. (2022) and also employed by Bavarian et al. (2022).”
The Stack: 3 TB of permissively licensed source code (2022, background): “A few studies have used such decoder-only models for code-infilling tasks via a causal masking mechanism (Fried et al., 2022; Bavarian et al., 2022).”
DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation (2022, background): “…that 1) infilling is an important functionality for real-world programming and 2) there is a growing trend in pre-training with the right context (Agha-janyan et al., 2022; Fried et al., 2022; Bavarian et al., 2022; Tay et al., 2022), we expect more future pre-trained models to perform infilling.”
Lost at C: A User Study on the Security Implications of Large Language Model Code Assistants (2022, background): “The last LLM is capable of filling in the middle given a prefix and suffix [48]).”
DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence (2024, unknown): “This demonstrates the superior effectiveness of DeepSeek-Coder-V2-Lite-Base, particularly in handling FIM tasks across different programming languages, achieving comparable performance with other bigger models in the evaluation.”
Qwen2.5-Coder Technical Report (2024, methodology): “5-Coder utilizes the Fill-In-the-Middle (FIM) training strategy, as introduced in Bavarian et al. (2022), enabling the model to generate code that is contextually coherent.”
CodeGen2: Lessons for Training LLMs on Programming and Natural Languages (2023, methodology): “Several variants (Du et al., 2022; Fried et al., 2022; Bavarian et al., 2022) Data Distribution In maximum likelihood learning, a model is learned by minimizing some measure of divergence between the data and model distribution.”
GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models (2025, unknown): “Moreover, the Fill-In-the-Middle [5] training objective is applied to all source code data.”
Large Language Diffusion Models (2025, unknown): “Code generation: HumanEval [120], HumanEval-FIM [121] and MBPP [122].”
DeepSeek-Coder: When the Large Language Model Meets Programming - The Rise of Code Intelligence (2024, methodology): “In addition to employing the next token prediction loss during pre-training, we have also incorporated the Fill-In-Middle (FIM) approach (Bavarian et al., 2022; Li et al., 2023).”
Scaling Laws for Generative Mixed-Modal Language Models (2023, background): “Recent work also shows that this masking does not hurt language modeling performance or the generative capacity of the models (Fried et al., 2022; Bavarian et al., 2022).”
Continuous diffusion for categorical data (2022, background): “15Infilling is also possible with autoregressive models if they are specifically trained for this task (Bavarian et al., 2022).”
FMI_SU_Yotkova_Kastreva at SemEval-2026 Task 13: Lightweight Detection of LLM-Generated Code via Stylometric Signals (2026, unknown): “This has led to increased presence of machine-generated code in cases such as automated unit test generation (Jain et al., 2025), code infilling and completion (Bavarian et al., 2022), and production development workflows (Dunay et al., 2024; Froemmgen et al., 2024).”