Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution

By Aaron Lou, Chenlin Meng, Stefano Ermon. International Conference on Machine Learning, 2023

651 citations recorded by the source corpus.

Despite their groundbreaking performance for many generative modeling tasks, diffusion models have fallen short on discrete data domains such as natural language. Crucially, standard diffusion models rely on the well-established theory of score matching, but efforts to generalize this to discrete structures have not yielded the same empirical gains. In this work, we bridge this gap by proposing score entropy, a novel loss that naturally extends score matching to discrete spaces, integrates seamlessly to build discrete diffusion models, and significantly boosts performance. Experimentally, we test our Score Entropy Discrete Diffusion models (SEDD) on standard language modeling tasks. For comparable model sizes, SEDD beats existing language diffusion paradigms (reducing perplexity by $25$-$75$\%) and is competitive with autoregressive models, in particular outperforming GPT-2. Furthermore, compared to autoregressive mdoels, SEDD generates faithful text without requiring distribution annealing techniques like temperature scaling (around $6$-$8\times$ better generative perplexity than un-annealed GPT-2), can trade compute and quality (similar quality with $32\times$ fewer network evaluations), and enables controllable infilling (matching nucleus sampling quality while enabling other strategies besides left to right prompting).

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/ce806f8d32f6fb1eaa821248a1bc4fa2cd949fbb

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jiacheng Sun
2. Kaiwen Xue
3. Shen Nie
4. Jingyang Ou
5. Fengqi Zhu
6. Chongxuan Li

In Computer Vision and Pattern Recognition

1. Jiacheng Sun
2. Kaiwen Xue
3. Shen Nie
4. Jingyang Ou
5. Fengqi Zhu
6. Zhenguo Li
7. Chongxuan Li
8. Jun Zhu

Citations in the corpus

• BLOCK DIFFUSION: INTERPOLATING BETWEEN AU-TOREGRESSIVE AND DIFFUSION LANGUAGE MODELS (2025, neutral): “Our work applies to extensions of D3PM (He et al., 2022; Lou et al., 2024) including ones in continuous time (Campbell et al., 2022; Sun et al., 2022).”
• Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data (2024, positive): “To alleviate this issue, existing work (Campbell et al., 2022; Lou et al., 2024) assumes independence between dimensions and each dimension is a one-dimensional diffusion process with the same transition rate matrix Q tok t ∈ R n × n .”