Simple and Effective Masked Diffusion Language Models

By Subham Sekhar Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin T. Chiu, Alexander Rush, Volodymyr Kuleshov. Neural Information Processing Systems, 2024

763 citations recorded by the source corpus.

While diffusion models excel at generating high-quality images, prior work reports a significant performance gap between diffusion and autoregressive (AR) methods in language modeling. In this work, we show that simple masked discrete diffusion is more performant than previously thought. We apply an effective training recipe that improves the performance of masked diffusion models and derive a simplified, Rao-Blackwellized objective that results in additional improvements. Our objective has a simple form -- it is a mixture of classical masked language modeling losses -- and can be used to train encoder-only language models that admit efficient samplers, including ones that can generate arbitrary lengths of text semi-autoregressively like a traditional language model. On language modeling benchmarks, a range of masked diffusion models trained with modern engineering practices achieves a new state-of-the-art among diffusion models, and approaches AR perplexity. We provide the code, along with a blog post and video tutorial on the project page: https://s-sahoo.com/mdlm

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/f8d357d38bbcdd93889fe71762eb57842b2ab063

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Shen Nie
2. Jingyang Ou
3. Fengqi Zhu
4. Chongxuan Li
5. Yankai Lin
6. Zhiyuan Liu
7. Maosong Sun

Citations in the corpus

• BLOCK DIFFUSION: INTERPOLATING BETWEEN AU-TOREGRESSIVE AND DIFFUSION LANGUAGE MODELS (2025, neutral): “We also qualitatively examine samples taken from BD3-LM and baselines (AR, MDLM) trained on the OWT dataset; we report samples in Suppl.”
• Large Language Diffusion Models (2025, positive): “Shi et al. [18], Sahoo et al. [19], Ou et al. [20] established fundamental theoretical results, which motivated our model design, training, and inference (see Appendix A for details).”
• Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding (2025, neutral): “More recently, work by MDLM [28, 27, 41] and RADD [22] has shown that for MDMs, different parameterizations are equivalent.”
• Dream 7B: Diffusion Large Language Models (2025, neutral): “Building on this foundation, several studies have improved the training objectives for discrete diffusion (Zheng et al., 2024b; Lou et al., 2024; Shi et al., 2024; Ou et al., 2025; Zhao et al., 2024; Sahoo et al., 2024).”