BLOCK DIFFUSION: INTERPOLATING BETWEEN AU-TOREGRESSIVE AND DIFFUSION LANGUAGE MODELS

By Marianne Arriola, Aaron Gokaslan, Justin T. Chiu, Zhihan Yang, Zhi-Hong Qi, Jiaqi Han, Subham Sekhar Sahoo, Volodymyr Kuleshov. International Conference on Learning Representations, 2025

334 citations recorded by the source corpus.

Diffusion language models offer unique benefits over autoregressive models due to their potential for parallelized generation and controllability, yet they lag in likelihood modeling and are limited to fixed-length generation. In this work, we introduce a class of block diffusion language models that interpolate between discrete denoising diffusion and autoregressive models. Block diffusion overcomes key limitations of both approaches by supporting flexible-length generation and improving inference efficiency with KV caching and parallel token sampling. We propose a recipe for building effective block diffusion models that includes an efficient training algorithm, estimators of gradient variance, and data-driven noise schedules to minimize the variance. Block diffusion sets a new state-of-the-art performance among diffusion models on language modeling benchmarks and enables generation of arbitrary-length sequences. We provide the code1, along with the model weights and blog post on the project page: https://m-arriola.com/bd3lms

View Paper: https://www.semanticscholar.org/paper/5312fec64ae16cd2a32bdc8c1fa6605bf77333ed

Citations in the corpus

• Dream 7B: Diffusion Large Language Models (2025, unknown): “Moreover, combining diffusion and autoregressive methods through block generation has emerged as a promising approach, demonstrating a flexible trade-off between parallelism and sequential coherence (Ye et al., 2024; Arriola et al., 2025).”
• Large Language Diffusion Models (2025, unknown): “In particular, it supports autoregressive and block diffusion [31] sampling directly after the pre-training or SFT processes described above, without requiring any further modifications or training.”
• Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion (2025, unknown): “Very recently, hybrid models integrating autoregressive and diffusion frameworks have emerged as a promising direction in generative modeling of videos [8, 16, 20, 22, 28, 33, 45, 50, 52, 89, 100, 106, 107] as well as other sequence domains [1, 12, 43, 53, 59, 90, 110].”