Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data
By Jingyang Ou, Shen Nie, Kaiwen Xue, Fengqi Zhu, Jiacheng Sun, Zhenguo Li, Chongxuan Li. International Conference on Learning Representations, 2024
345 citations recorded by the source corpus.
Discrete diffusion models with absorbing processes have shown promise in language modeling. The key quantities to be estimated are the ratios between the marginal probabilities of two transitive states at all timesteps, called the concrete score. In this paper, we reveal that the concrete score in absorbing diffusion can be expressed as conditional probabilities of clean data, multiplied by a time-dependent scalar in an analytic form. Motivated by this finding, we propose reparameterized absorbing discrete diffusion (RADD), a dedicated diffusion model without time-condition that characterizes the time-independent conditional probabilities. Besides its simplicity, RADD can reduce the number of function evaluations (NFEs) by caching the output of the time-independent network when the noisy sample remains unchanged in a sampling interval, which enables sampling acceleration. Built upon the new perspective of conditional distributions, we further unify absorbing discrete diffusion and any-order autoregressive models (AO-ARMs), showing that the upper bound on the negative log-likelihood for the diffusion model can be interpreted as an expected negative log-likelihood for AO-ARMs. Further, our RADD models achieve SOTA performance among diffusion models on 5 zero-shot language modeling benchmarks (measured by perplexity) at the GPT-2 scale. Our code is available at https://github.com/ML-GSAI/RADD.
View Paper: https://www.semanticscholar.org/paper/7a73cc4bc0ce80661b5acbb51bbde30ea5167280
Citations in the corpus
- Simplified and Generalized Masked Diffusion for Discrete Data (2024, unknown): “The relation between our simplified ELBO (5) and the AO-ARM objective is independently clarified by Ou et al. [36].”
- Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding (2025, unknown): “More recently, work by MDLM [28, 27, 41] and RADD [22] has shown that for MDMs, different parameterizations are equivalent.”
- Large Language Diffusion Models (2025, unknown): “It is trained using a cross-entropy loss computed only on the masked tokens [18–20]: where x 0 is a training sample, t is a continuous random variable drawn uniformly from [0 , 1] , x t is sampled from the forward process and L is the sequence length.”
- Simple and Effective Masked Diffusion Language Models (2024, unknown): “…et al. [1] show that the posterior corresponding to (13) is given as follows: which we simplify to the following: q ( z s | z t , x ) A.1.3 p θ ( z s | z t ) Austin et al. [1] approximate the reverse process in the following manner: where x θ ( z t ,t ): V× [0 , 1] → ∆ K is an approximation for x .”
- Dream 7B: Diffusion Large Language Models (2025, unknown): “Building on this foundation, several studies have improved the training objectives for discrete diffusion (Zheng et al., 2024b; Lou et al., 2024; Shi et al., 2024; Ou et al., 2025; Zhao et al., 2024; Sahoo et al., 2024).”
- BLOCK DIFFUSION: INTERPOLATING BETWEEN AU-TOREGRESSIVE AND DIFFUSION LANGUAGE MODELS (2025, unknown): “Here, we introduce masked BD3-LMs, a special class of block diffusion models based on the masked diffusion language modeling framework (Sahoo et al., 2024a; Shi et al., 2024; Ou et al., 2025).”