Structured Denoising Diffusion Models in Discrete State-Spaces

By Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, Rianne van den Berg. 2021

1,984 citations recorded by the source corpus.

Denoising diffusion probabilistic models (DDPMs) (Ho et al. 2020) have shown impressive results on image and waveform generation in continuous state spaces. Here, we introduce Discrete Denoising Diffusion Probabilistic Models (D3PMs), diffusion-like generative models for discrete data that generalize the multinomial diffusion model of Hoogeboom et al. 2021, by going beyond corruption processes with uniform transition probabilities. This includes corruption with transition matrices that mimic Gaussian kernels in continuous space, matrices based on nearest neighbors in embedding space, and matrices that introduce absorbing states. The third allows us to draw a connection between diffusion models and autoregressive and mask-based generative models. We show that the choice of transition matrix is an important design decision that leads to improved results in image and text domains. We also introduce a new loss function that combines the variational lower bound with an auxiliary cross entropy loss. For text, this model class achieves strong results on character-level text generation while scaling to large vocabularies on LM1B. On the image dataset CIFAR-10, our models approach the sample quality and exceed the log-likelihood of the continuous-space DDPM model.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models, Model Architectures, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

In Model Architectures

In Signal Processing

Citations in the corpus

Diffusion Autoencoders: Toward a Meaningful and Decodable Representation (2021, background): “Numerous applications have emerged notably in the image domain, such as image manipulation, translation, super-resolution [8,32,35,43], in speech and text domains [5,6], or 3D point cloud [34].”
Protein Structure and Sequence Generation with Equivariant Denoising Diffusion Probabilistic Models (2022, methodology): “We use an approach akin to a masked language model [12] to generate the sequences on top of the backbones, which can be interpreted as diffusion with an absorbing state [5].”
Palette: Image-to-Image Diffusion Models (2021, methodology): “2015; Song and Ermon 2020] have received a surge of recent interest [Austin et al. 2021; Cai et al. 2020; Hoogeboom et al. 2021; Kingma et al. 2021; Song et al. 2021; Vahdat et al. 2021], resulting in several key advances in modeling continuous data.”
InstructTTS: Modelling Expressive TTS in Discrete Latent Space With Natural Language Style Prompt (2023, methodology): “The more details about discrete diffusion models, please refer to [23], [34], [45].”
Diffusion-LM Improves Controllable Text Generation (2022, background): “, each token has some probability to be corrupted to an absorbing or random token) [1, 15, 16].”
Show-o: One Single Transformer to Unify Multimodal Understanding and Generation (2024, neutral): “In contrast, D3PM (Austin et al., 2021), Mask-predict (Ghazvininejad et al., 2019), ARDM (Hoogeboom et al., 2022) and MaskGIT (Chang et al., 2022) formulate a discrete corruption process as an alternative to Gaussian diffusion.”
Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioning (2022, methodology): “Due to the discreteness, existing work has to re-formulate continuous diffusion models by adopting a discrete data space and state space (Sohl-Dickstein et al., 2015; Hoogeboom et al., 2021; Austin et al., 2021).”
Step-unrolled Denoising Autoencoders for Text Generation (2021, background): “Yet, recently the community has seen promising results from non-AR models like Multinomial Diffusion (Hooge-boom et al., 2021) and D3PM (Austin et al., 2021).”
MAGVIT: Masked Generative Video Transformer (2022, background): “For example, the state-of-the-art video diffusion model [33] extends the image denoising diffusion model [3,32,52,54,60] by incorporating 3D U-Net [14] architectures and joint training on both images and videos.”
KNN-Diffusion: Image Generation via Large-Scale Retrieval (2022, background): “[1] introduced a novel Discrete Denoising Diffusion Probabilistic Models (D3PMs), which generalizes the multinomial diffusion model of Hoogeboom et al.”
BLOCK DIFFUSION: INTERPOLATING BETWEEN AU-TOREGRESSIVE AND DIFFUSION LANGUAGE MODELS (2025, neutral): “Our work applies to extensions of D3PM (He et al., 2022; Lou et al., 2024) including ones in continuous time (Campbell et al., 2022; Sun et al., 2022).”
Itô-Taylor Sampling Scheme for Denoising Diffusion Probabilistic Models using Ideal Derivatives (2021, background): “…impressive results in many ﬁelds including image generation (Ho et al., 2020; Vahdat et al., 2021), image superresoution (Saharia et al., 2021; Ho et al., 2021), image translation (Sasaki et al., 2021), speech synthesis (Chen et al., 2020, 2021; Kong et al., 2021; Popov et al., 2021),…”
TabDDPM: Modelling Tabular Data with Diffusion Models (2022, methodology): “Beyond computer vision, the DDPM framework is also investigated in other fields, such as NLP (Austin et al., 2021; Li et al., 2022), waveform signal processing (Kong et al., 2020; Chen et al., 2020b), molecular graphs (Jing et al., 2022; Hooge-boom et al., 2022), time series (Tashiro et al., 2021), ”
Compositional Visual Generation with Composable Diffusion Models (2022, background): “While diffusion models have been developed for image generation [39], they have further proven successful in the generation of waveforms [3], 3D shapes [47], decision making [16] and text [1], suggesting that our proposed composition operators may further be applied in such domains.”
Score-Based Generative Modeling with Critically-Damped Langevin Diffusion (2021, result): “Furthermore, our model reaches an NLL bound of 3.31, which is on par with recent works such as Nichol & Dhariwal (2021); Austin et al. (2021); Vahdat et al. (2021) and indicates that our model is not dropping modes.”
A Survey on Generative Diffusion Models (2022, methodology): “D3PM [16] deﬁnes the forward process in discrete space, processing data like text or atom type, using transition kernels Q t : where Cat() denotes a categorical distribution.”