Score-Based Generative Modeling with Critically-Damped Langevin Diffusion

By Tim Dockhorn, Arash Vahdat, Karsten Kreis. 2021

286 citations recorded by the source corpus.

Score-based generative models (SGMs) have demonstrated remarkable synthesis quality. SGMs rely on a diffusion process that gradually perturbs the data towards a tractable distribution, while the generative model learns to denoise. The complexity of this denoising task is, apart from the data distribution itself, uniquely determined by the diffusion process. We argue that current SGMs employ overly simplistic diffusions, leading to unnecessarily complex denoising processes, which limit generative modeling performance. Based on connections to statistical mechanics, we propose a novel critically-damped Langevin diffusion (CLD) and show that CLD-based SGMs achieve superior performance. CLD can be interpreted as running a joint diffusion in an extended space, where the auxiliary variables can be considered “velocities” that are coupled to the data variables as in Hamiltonian dynamics. We derive a novel score matching objective for CLD and show that the model only needs to learn the score function of the conditional distribution of the velocity given data, an easier task than learning scores of the data directly. We also derive a new sampling scheme for efﬁcient synthesis from CLD-based diffusion models. We ﬁnd that CLD outperforms previous SGMs in synthesis quality for similar network architectures and sampling compute budgets. We show that our novel sampler for CLD signiﬁcantly outperforms solvers such as Euler–Maruyama. Our framework provides new insights into score-based denoising diffusion models and can be readily used for high-resolution image synthesis. VPSDE. We leave the study of CLD with maximum likelihood training for high-dimensional (image) datasets to future work.

Subfields: Computational Theory and Mathematics, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Generative Models, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Theory and Mathematics

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Generative Models

In Probabilistic & Bayesian Methods

Citations in the corpus

Elucidating the Design Space of Diffusion-Based Generative Models (2022, methodology): “Improving the output quality and/or decreasing the computational cost of sampling are common topics in diffusion model research (e.g., [10, 24, 30, 31, 32, 36, 43, 50, 52, 53, 56]).”
DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps (2022, background): “For instance, defining a low-dimensional latent variable for DPMs [34]; designing special diffusion processes with bounded score functions [35]; combining GANs with the reverse process of DPMs [36].”
Diffusion Models in Vision: A Survey (2022, methodology): “[21] image generation unconditional NCSN++, DDPM++ CIFAR-10”
Subspace Diffusion Generative Models (2022, methodology): “The sole exception is the Langevin diffusion in CLD-SGM [4], but this required new forms of score-matching and specialised SDE solvers for numerical stability.”
Itô-Taylor Sampling Scheme for Denoising Diffusion Probabilistic Models using Ideal Derivatives (2021, methodology): “This is a common issue in DDPM studies, and there have been some studies aiming at improving the framework so that efﬁcient sampling is possible (Song et al., 2020a; Nichol & Dhariwal, 2021; Dockhorn et al., 2021).”
Align Your Latents: High-Resolution Video Synthesis with Latent Diffusion Models (2023, background): “However, despite advances in model distillation [49,52,73] and accelerated sampling [4,11,12,36,45,46,80,96,100,108], DMs generally require repeated evaluations of a computationally demanding large U-Net.”
Soft Truncation: A Universal Training Technique of Score-based Diffusion Model for High Precision Score Estimation (2021, methodology): “It is the common practice of continuous diffusion models (Song et al., 2021b;a; Dockhorn et al., 2022) to report their performances with log p θ ( x 0 ) , but Kim et al. (2022) show that log p θ ( x (cid:15) ) differs to log p θ ( x 0 ) by 0.05 in BPD scale when (cid:15) = 10 − 5 , which is quite…”
Diffusion Models: A Comprehensive Survey of Methods and Applications (2022, methodology): “To obtain the time reversal of the extended SDE, CLD only needs to learn the score function of the conditional distribution of velocity given data, arguably easier than learning scores of data directly.”
Fast Sampling of Diffusion Models with Exponential Integrator (2022, methodology): “We notice t 0 has a huge inﬂuence on image FIDs, which is also noticed and investigated across different studies (Kim et al., 2021; Dockhorn et al., 2021).”
A Continuous Time Framework for Discrete Denoising Models (2022, background): “[18] Tim Dockhorn, Arash Vahdat, and Karsten Kreis.”
Estimating the Optimal Covariance with Imperfect Mean in Diffusion Probabilistic Models (2022, unknown): “…with its FID results in Table 2 roots in the different natures of the two metrics, which has been observed and explored extensively in prior works (Ho et al., 2020; Nichol & Dhari-wal, 2021; Song et al., 2021; Vahdat et al., 2021; Watson et al., 2021; Kingma et al., 2021; Bao et al., 2022).”
Score-based Generative Modeling of Graphs via the System of Stochastic Differential Equations (2022, methodology): “Thus we propose a novel integrator, Symmetric Splitting for System of SDEs (S4) to simulate the system of reverse-time SDEs, that is efficient yet accurate, inspired by the Symmetric Splitting CLD Sampler (SSCS) (Dockhorn et al., 2021) and the Predictor-Corrector Sampler (PC sampler) (Song et al., 2”
UniPC: A Unified Predictor-Corrector Framework for Fast Sampling of Diffusion Models (2023, methodology): “Fast samplers of DPMs can be either training-based [31, 2, 37] or training-free [25, 26, 40, 24, 41].”
SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformers (2024, methodology): “Velocity field parameterizations using the Linear interpolant were also studied in [39, 42], and were generalized to the manifold setting in [6].”
Masked Diffusion Transformer is a Strong Image Synthesizer (2023, background): “Additionally, some training schemes [2,12] are introduced to improve the diffusion model training, e.g ., approximate maximum likelihood training [28,34,47], and training loss weighting [26,27].”