Maximum Likelihood Training of Score-Based Diffusion Models

By Yang Song, Conor Durkan, Iain Murray, Stefano Ermon. 2021

923 citations recorded by the source corpus.

Score-based diffusion models synthesize samples by reversing a stochastic process that diffuses data to noise, and are trained by minimizing a weighted combination of score matching losses. The log-likelihood of score-based diffusion models can be tractably computed through a connection to continuous normalizing flows, but log-likelihood is not directly optimized by the weighted combination of score matching losses. We show that for a specific weighting scheme, the objective upper bounds the negative log-likelihood, thus enabling approximate maximum likelihood training of score-based diffusion models. We empirically observe that maximum likelihood training consistently improves the likelihood of score-based diffusion models across multiple datasets, stochastic processes, and model architectures. Our best models achieve negative log-likelihoods of 2.83 and 3.76 bits/dim on CIFAR-10 and ImageNet 32x32 without any data augmentation, on a par with state-of-the-art autoregressive models on these tasks.

Subfields: Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Statistical and Nonlinear Physics, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

In Statistical and Nonlinear Physics

In Generative Models

Citations in the corpus

Classifier-Free Diffusion Guidance (2022, background): “…recently emerged as an expressive and ﬂexible family of generative models, delivering competitive sample quality and likelihood scores on image and audio synthesis tasks (Sohl-Dickstein et al., 2015; Song & Ermon, 2019; Ho et al., 2020; Song et al., 2021b; Kingma et al., 2021; Song et al., 2021a).”
Variational Diffusion Models (2021, methodology): “This would have been state-of-the-art until recently, but is worse than recent diffusion models that specifically target FID scores [Nichol and Dhariwal, 2021, Song et al., 2021b, Ho et al., 2020].”
Score-based Generative Modeling in Latent Space (2021, methodology): “[4] developed a bound for the data likelihood in their Theorem 3 of their second version, using a denoising score matching objective, closely related to our cross entropy expression.”
Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Models (2022, background): “Such a behavior essentially roots in the different natures of the two metrics and has been investigated in extensive prior works (Theis et al., 2015; Ho et al., 2020; Nichol & Dhariwal, 2021; Song et al., 2021; Vahdat et al., 2021; Watson et al., 2021; Kingma et al., 2021).”
DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps (2022, background): “λ is equivalent to the objectives in [10, 41], and the exact solution of the diffusion ODEs are proposed in Proposition 3.”
Flow Matching for Generative Modeling (2022, positive): “…Matching (SM) loss from Song & Ermon (2019), while considering λ ( t ) = β (1 − t ) ( β is defined below) corresponds to the Score Flow (SF) loss motivated by an NLL upper bound (Song et al., 2021) When computing FID/Inception scores for CIFAR10, ImageNet-32/64 we use the TensorFlow GAN library 2 .”
Continuous diffusion for categorical data (2022, methodology): “In fact, the most commonly used modelling setup for images implicitly reduces the loss weighting of high frequency content relative to likelihood-based models, allowing for a more eﬃcient use of model capacity which is well aligned with human perception (Song et al., 2021a).”
Diffusion Probabilistic Modeling for Video Generation (2022, background): “Recently, diffusion probabilistic models [51, 22, 53] have achieved considerable progress in image generation, with perceptual qualities comparable to GANs while having the benefit of being likelihoodbased [22, 52].”
Diffusion Models for Adversarial Purification (2022, background): “The proof follows (Song et al., 2021a; Lyu, 2009) that build connections between Fisher divergence and the “rate of change” in KL divergence by generalizing the de Bruijn’s identity (Barron, 1986), which we defer to Appendix A.1.”
Fast Sampling of Diffusion Models with Exponential Integrator (2022, background): “…Rombach et al., 2021; Dhariwal & Nichol, 2021), text conditioned image generation (Nichol et al., 2021; Ramesh et al., 2022), text generation (Hoogeboom et al., 2021; Austin et al., 2021), 3D point cloud generation (Lyu et al., 2021), inverse problem (Kawar et al., 2021; Song et al., 2021b), etc.”
Learning Fast Samplers for Diffusion Models by Differentiating Through Sample Quality (2022, background): “Song et al. (2021b) derive a reverse SDE that, when discretized, uses different coefficients than the ancestral samplers considered in this work.”
Masked Diffusion Transformer is a Strong Image Synthesizer (2023, background): “Additionally, some training schemes [2,12] are introduced to improve the diffusion model training, e.g ., approximate maximum likelihood training [28,34,47], and training loss weighting [26,27].”
Retrieval-Augmented Generation for AI-Generated Content: A Survey (2024, neutral): “In this section, we introduce RAG foundations and outline enhancement methods that further improve the effectiveness.”
Soft Truncation: A Universal Training Technique of Score-based Diffusion Model for High Precision Score Estimation (2021, methodology): “Henceforth, this paper discusses the argument of truncating diffusion time (VPSDE) and diffusion variance (VESDE) exchangeably.”
Estimating the Optimal Covariance with Imperfect Mean in Diffusion Probabilistic Models (2022, result): “…with its FID results in Table 2 roots in the different natures of the two metrics, which has been observed and explored extensively in prior works (Ho et al., 2020; Nichol & Dhari-wal, 2021; Song et al., 2021; Vahdat et al., 2021; Watson et al., 2021; Kingma et al., 2021; Bao et al., 2022).”
Score-Based Generative Modeling with Critically-Damped Langevin Diffusion (2021, methodology): “(56) Note again that Lt is different for HSM and DSM. Analogously to prior work (Ho et al., 2020; Vahdat et al., 2021; Song et al., 2021b) an objective better suited for high quality image synthesis can be obtained by “dropping the variance prefactor”: HSM ( λ(t) = ( `HSMt )−2) = Et∼U…”
Building Normalizing Flows with Stochastic Interpolants (2022, positive): “We stress however that while our velocity is well-behaved for all times in [0 , 1] , as shown in (B.20), the drift and diffusion coefﬁcient in the associated SDE are singular at t = 0 , 1 .”
Consistency Trajectory Models: Learning Probability Flow ODE Trajectory of Diffusion (2023, methodology): “Additionally, CTM’s ability to approximate scores using gθ(xt, t, t) enables evaluating Negative Log-Likelihood (NLL) (Song et al., 2021; Kim et al., 2022b), establishing a new SOTA NLL.”