A Variational Perspective on Diffusion-Based Generative Models and Score Matching

By Chin-Wei Huang, Jae Hyun Lim, Aaron C. Courville. 2021

239 citations recorded by the source corpus.

Discrete-time diffusion-based generative models and score matching methods have shown promising results in modeling high-dimensional image data. Recently, Song et al. (2021) show that diffusion processes that transform data into noise can be reversed via learning the score function, i.e. the gradient of the log-density of the perturbed data. They propose to plug the learned score function into an inverse formula to define a generative diffusion process. Despite the empirical success, a theoretical underpinning of this procedure is still lacking. In this work, we approach the (continuous-time) generative diffusion directly and derive a variational framework for likelihood estimation, which includes continuous-time normalizing flows as a special case, and can be seen as an infinitely deep variational autoencoder. Under this framework, we show that minimizing the score-matching loss is equivalent to maximizing a lower bound of the likelihood of the plug-in reverse SDE proposed by Song et al. (2021), bridging the theoretical gap.

Subfields: Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Generative Models

Citations in the corpus

Score-based Generative Modeling in Latent Space (2021, background): “Moreover, [25] discusses the likelihood-based training of SGMs from a fundamental perspective and shows how several score matching objectives become a variational bound on the data likelihood.”
Elucidating the Design Space of Diffusion-Based Generative Models (2022, background): “The SDEs of Song et al. [48] can be generalized [20, 55] as a sum of the probability flow ODE of Eq.”
StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models (2023, methodology): “We sample s by EDM [50] that follows the combined probability flow [51] and time-varying Langevin dynamics [52]:”
Tackling the Generative Learning Trilemma with Denoising Diffusion GANs (2021, background): “They also obtain good mode coverage, indicated by high likelihood (Song et al., 2021b; Kingma et al., 2021; Huang et al., 2021).”
Learning Fast Samplers for Diffusion Models by Differentiating Through Sample Quality (2022, background): “Huang et al. (2021) generalize this family of samplers to a “plug-in reverse SDE” that interpolates between a probability flow ODE and the reverse SDE, similarly to how the DDIM η interpolates between an implicit probabilistic model and a stochastic reverse process.”
Riemannian Score-Based Generative Modeling (2022, methodology): “Finally, we note that there exist other approaches to introduce SGMs using variational and maximum likelihood techniques (Ho et al., 2020; Huang et al., 2021; Durkan and Song, 2021).”
A Survey on Generative Diffusion Models (2022, methodology): “1) MLE Training: Three concurrent works–ScoreFlow [103], VDM [77], and [104] establish a connection between the MLE training and the weighted denoising score-matching (DSM) objective in diffusion models, primarily through the use of the Girsanov theorem.”
SE(3) diffusion model with application to protein backbone generation (2023, methodology): “Concurrently, Huang et al. (2022) introduced a similar framework extending the maximum likelihood approach of Huang et al. (2021).”
Come-Closer-Diffuse-Faster: Accelerating Conditional Diffusion Models for Inverse Problems through Stochastic Contraction (2021, background): “For the given forward SDE in (1), there exists a reversetime SDE running backwards [12, 34]:”
SegDiff: Image Segmentation with Diffusion Probabilistic Models (2021, methodology): “A variational framework for the likelihood estimation of diffusion models was introduced by Huang et al. [21].”
Soft Truncation: A Universal Training Technique of Score-based Diffusion Model for High Precision Score Estimation (2021, background): “Previous works (Song et al., 2021a; Huang et al., 2021) apply the importance sampling with the importance distribution of (cid:107) − ∇ | , but sampling of Monte-Carlo diffusion time from p ∗ iw ( t ) at every training iteration would incur 2 slower training speed, at least, because the importance…”
Score Matching Model for Unbounded Data Score (2021, methodology): “Recent papers [19, 6, 20, 12, 13, 21] apply the importance sampling to minimize the estimation variance by changing the weight function of L(θ;λ, ) from λ(t) to a new weight function, λ̃(t).”
Understanding Diffusion Objectives as the ELBO with Simple Data Augmentation (2023, background): “Earlier work [Kingma et al., 2021, Song et al., 2021a, Huang et al., 2021, Vahdat et al., 2021], including Variational Diffusion Models [Kingma et al., 2021], showed how to optimize continous-time diffusion models towards the ELBO objective.”
Score-Based Generative Modeling with Critically-Damped Langevin Diffusion (2021, background): “Some works study SGMs for maximum likelihood training (Song et al., 2021b; Kingma et al., 2021; Huang et al., 2021).”
Speech Enhancement and Dereverberation With Diffusion-Based Generative Models (2022, unknown): “obtain the so-called plug-in reverse SDE [40],”
Diffusion Models: A Comprehensive Survey of Methods and Applications (2022, methodology): “Studies have focused on maximizing the likelihood of the solution of above SDE and ODE at time 0, denoted by p 0 and p 0 , respectively [87, 128, 190].”