Continuous diffusion for categorical data

By Sander Dieleman, Laurent Sartran, Arman Roshannai, Nikolay Savinov, Yaroslav Ganin, Pierre H. Richemond, Arnaud Doucet, Robin Strudel, Chris Dyer, Conor Durkan and 4 more. 2022

209 citations recorded by the source corpus.

Diffusion models have quickly become the go-to paradigm for generative modelling of perceptual signals (such as images and sound) through iterative refinement. Their success hinges on the fact that the underlying physical phenomena are continuous. For inherently discrete and categorical data such as language, various diffusion-inspired alternatives have been proposed. However, the continuous nature of diffusion models conveys many benefits, and in this work we endeavour to preserve it. We propose CDCD, a framework for modelling categorical data with diffusion models that are continuous both in time and input space. We demonstrate its efficacy on several language modelling tasks.

Subfields: Artificial Intelligence, Generative Models, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Generative Models

In Signal Processing

Citations in the corpus

Scalable Adaptive Computation for Iterative Generation (2022, methodology): “, 2021) and has been adopted in several existing work (Strudel et al., 2022; Dieleman et al., 2022; Chen et al., 2022a).”
Diffusion Models: A Comprehensive Survey of Methods and Applications (2022, methodology): “CDCD [52] proposes to model categorical data (including texts) with diffusion models that are continuous both in time and input space, and designs a score interpolation technique for optimization.”
Large Language Diffusion Models (2025, unknown): “Alternatively, some methods model continuous parameters of discrete distributions instead [52– 56].”
Dream 7B: Diffusion Large Language Models (2025, unknown): “Building on their remarkable success, a number of works have extended diffusion techniques to text generation tasks by modeling text in a continuous embedding space, building continuous diffusion language models (Li et al., 2022; Gong et al., 2023b,a; Dieleman et al., 2022).”
BLOCK DIFFUSION: INTERPOLATING BETWEEN AU-TOREGRESSIVE AND DIFFUSION LANGUAGE MODELS (2025, unknown): “Comparison to Gaussian Diffusion Alternatively, one may perform diffusion over continuous embeddings of discrete tokens (Li et al., 2022; Dieleman et al., 2022; Chen et al., 2022).”
Simplified and Generalized Masked Diffusion for Discrete Data (2024, unknown): “This has motivated the development of continuous space diffusion models where the discrete data are embedded in the Euclidean space [19, 20, 21, 22, 23] or the simplex [24, 25, 26, 27, 28].”
Discrete Flow Matching (2024, methodology): “Another line of works has been exploring the use of continuous space diffusion for discrete data, typically operating in the logits space (Dieleman et al., 2022; Li et al., 2022; Han et al., 2022; Lin et al., 2022; Chen et al., 2022).”
DIFUSCO: Graph-based Diffusion Solvers for Combinatorial Optimization (2023, background): “Another line of work studies diffusion models for discrete data by applying continuous diffusion models with Gaussian noise on the embedding space of discrete data [30, 69, 24], the {− 1 .”
Understanding Diffusion Objectives as the ELBO with Simple Data Augmentation (2023, unknown): “Similar noise schedules have been in Zheng et al. [2023], Nichol and Dhariwal [2021], Dieleman et al. [2022].”
Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution (2023, background): “…prior works improve downstream performance with alternative loss functions (moving away from likelihood-based score matching) (Han et al., 2022; Mahabadi et al., 2023) and explicitly encoding conditional information (e.g. inputting an infilling mask) (Gong et al., 2023; Dieleman et al., 2022).”