A Continuous Time Framework for Discrete Denoising Models

By Andrew Campbell, Joe Benton, Valentin De Bortoli, Tom Rainforth, George Deligiannidis, Arnaud Doucet. Neural Information Processing Systems, 2022

435 citations recorded by the source corpus.

We provide the first complete continuous time framework for denoising diffusion models of discrete data. This is achieved by formulating the forward noising process and corresponding reverse time generative process as Continuous Time Markov Chains (CTMCs). The model can be efficiently trained using a continuous time version of the ELBO. We simulate the high dimensional CTMC using techniques developed in chemical physics and exploit our continuous time framework to derive high performance samplers that we show can outperform discrete time methods for discrete data. The continuous time treatment also enables us to derive a novel theoretical result bounding the error between the generated sample distribution and the true data distribution.

Subfields: Computational Mathematics, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics

View Paper: https://www.semanticscholar.org/paper/e9b9a47cd81c66603c827f0f2bc4fba0d9ae77c4

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mathematics

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

Citations in the corpus

Diffusion Models: A Comprehensive Survey of Methods and Applications (2022, background): “For example, studies have found that analyzing the continuous time diffusion model can yield better sample quality [7, 21].”
DIFUSCO: Graph-based Diffusion Solvers for Combinatorial Optimization (2023, methodology): “…on geometrical NP-complete combinatorial optimization problems such as Euclidean TSP. Finally, we are interested in utilizing (higher-order) accelerated inference techniques for diffusion model-based solvers, such as those inspired by the continuous time framework for discrete diffusion [12, 105].”
Show-o: One Single Transformer to Unify Multimodal Understanding and Generation (2024, neutral): “5”
Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data (2024, positive): “To alleviate this issue, existing work (Campbell et al., 2022; Lou et al., 2024) assumes independence between dimensions and each dimension is a one-dimensional diffusion process with the same transition rate matrix Q tok t ∈ R n × n .”
Dream 7B: Diffusion Large Language Models (2025, positive): “To fit the discrete nature of text, Austin et al. (2021a); Hoogeboom et al. (2021); Campbell et al. (2022) introduced discrete diffusion, which operates on the discrete vocabulary space, establishing the foundation for discrete diffusion methods.”
Discrete Flow Matching (2024, methodology): “…discrete setting has been made via mostly two approaches: embedding the discrete data in continuous space and applying continuous diffusion (Dieleman et al., 2022; Stark et al., 2024) or designing diffusion or flow processes over discrete state spaces (Austin et al., 2021a; Campbell et al., 2022).”
BLOCK DIFFUSION: INTERPOLATING BETWEEN AU-TOREGRESSIVE AND DIFFUSION LANGUAGE MODELS (2025, positive): “Our work applies to extensions of D3PM (He et al., 2022; Lou et al., 2024) including ones in continuous time (Campbell et al., 2022; Sun et al., 2022).”
A Survey on Generative Diffusion Models (2022, background): “2) Cross-Distribution Generation: Multimodal guidance has signiﬁcantly improved generative capabilities in medical analysis.”
Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution (2023, methodology): “In particular, the implicit version coincides with the likelihood loss introduced in Campbell et al. (2022).”
Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding (2025, positive): “Recent advancements in discrete diffusion models [2, 20, 21, 11, 3, 10, 18, 24, 31, 14, 42, 4, 37, 27, 28, 41, 5, 40, 39] have reshaped the landscape of text generation, offering a viable alternative to autoregressive (AR) paradigms in large language models (LLMs).”
Simple and Effective Masked Diffusion Language Models (2024, neutral): “However, extending D3PM in this way requires computing the limit of the product of an infinite number of matrices Q T which requires advanced CTMC theory [4].”
Simplified and Generalized Masked Diffusion for Discrete Data (2024, positive): “Notably, despite lacking knowledge of the ordinal structure of pixel values, MD4 outperforms models trained with this inductive bias, including D3PM Gauss and Campbell et al. [29] where the noising distribution is a discrete Gaussian that assigns larger probabilities to near pixel values.”
TabDDPM: Modelling Tabular Data with Diffusion Models (2022, neutral): “However, alternative approaches exist, e.g., (Chen et al., 2022; Campbell et al., 2022; Zheng & Charoenphakdee, 2022).”
DiGress: Discrete Denoising diffusion for graph generation (2022, background): “, 2022), as well as theoretically studied in continuous time settings (Campbell et al., 2022).”
Large Language Diffusion Models (2025, positive): “Austin et al. [16] was among the pioneering works that introduced discrete diffusion models into language modeling, demonstrating the feasibility of this approach.”
Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioning (2022, methodology): “We compare our model against state-of-the-art generative models (Ho et al., 2020; Austin et al., 2021; Campbell et al., 2022; Jun et al., 2020) on generating discrete C IFAR -10 images in Table 1.”
Continuous diffusion for categorical data (2022, background): “Campbell et al. (2022) and Sun et al. (2022) propose continuous-time models for discrete input, though they do not explore the application to language modelling.”