Understanding Diffusion Objectives as the ELBO with Simple Data Augmentation

By Diederik P. Kingma, Ruiqi Gao. Neural Information Processing Systems, 2023

325 citations recorded by the source corpus.

To achieve the highest perceptual quality, state-of-the-art diffusion models are optimized with objectives that typically look very different from the maximum likelihood and the Evidence Lower Bound (ELBO) objectives. In this work, we reveal that diffusion model objectives are actually closely related to the ELBO. Specifically, we show that all commonly used diffusion model objectives equate to a weighted integral of ELBOs over different noise levels, where the weighting depends on the specific objective used. Under the condition of monotonic weighting, the connection is even closer: the diffusion objective then equals the ELBO, combined with simple data augmentation, namely Gaussian noise perturbation. We show that this condition holds for a number of state-of-the-art diffusion models. In experiments, we explore new monotonic weightings and demonstrate their effectiveness, achieving state-of-the-art FID scores on the high-resolution ImageNet benchmark.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Management Science and Operations Research

View Paper: https://www.semanticscholar.org/paper/11c8911f787eed34e6f058d5d36287d45c54c961

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Management Science and Operations Research

Citations in the corpus

Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think (2024, unknown): “Specifically, we consider the following four types of approaches: (a) Pixel diffusion : ADM (Dhariwal & Nichol, 2021), VDM ++ (Kingma & Gao, 2024), Simple diffusion (Hoogeboom et al., 2023), CDM (Ho et al., 2022), (b) Latent diffusion with U-Net : LDM (Rombach et al., 2022), (c) Latent diffusion…”
FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Space (2025, unknown): “…our models, we construct forward noising processes in the latent space of an image autoencoder as with x 0 ∼ p data , ε ∼ N (0 , 1) , and the coefficients a t and b t define the log signal-to-noise ratio (log-SNR) [25] λ and we sample t from a Logit-Normal Distribution [12]: p ( t ) = exp( − 0 .”
F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching (2024, methodology): “…with x 0 , we have Further leveraging Optimal Transport form ψ t ( x ) = (1 − t ) x + tx 1 , we have the OT-CFM loss, To view in a more general way [43], if formulating the loss in terms of log signal-to-noise ratio (log-SNR) λ instead of ﬂow step t , and parameterizing to predict x 0 ( ǫ ,…”
Scaling Rectified Flow Transformers for High-Resolution Image Synthesis (2024, methodology): “Note that this choice results in The network is parameterized through an F -prediction (Kingma & Gao, 2023; Karras et al., 2022) and the loss can be written as Cosine (Nichol & Dhariwal, 2021) proposed a forward process of the form In combination with an ϵ -parameterization and loss, this…”
Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion (2024, unknown): “We speculate that the better result is due to the same data-augmentation effect described in prior works [39].”