Variational Diffusion Models

By Diederik P. Kingma, Tim Salimans, Ben Poole, Jonathan Ho. 2021

1,552 citations recorded by the source corpus.

Diffusion-based generative models have demonstrated a capacity for perceptually impressive synthesis, but can they also be great likelihood-based models? We answer this in the affirmative, and introduce a family of diffusion-based generative models that obtain state-of-the-art likelihoods on standard image density estimation benchmarks. Unlike other diffusion-based models, our method allows for efficient optimization of the noise schedule jointly with the rest of the model. We show that the variational lower bound (VLB) simplifies to a remarkably short expression in terms of the signal-to-noise ratio of the diffused data, thereby improving our theoretical understanding of this model class. Using this insight, we prove an equivalence between several models proposed in the literature. In addition, we show that the continuous-time VLB is invariant to the noise schedule, except for the signal-to-noise ratio at its endpoints. This enables us to learn a noise schedule that minimizes the variance of the resulting VLB estimator, leading to faster optimization. Combining these advances with architectural improvements, we obtain state-of-the-art likelihoods on image density estimation benchmarks, outperforming autoregressive models that have dominated these benchmarks for many years, with often significantly faster optimization. In addition, we show how to use the model as part of a bits-back compression scheme, and demonstrate lossless compression rates close to the theoretical optimum. Code is available at https://github.com/google-research/vdm .

Subfields: Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Generative Models, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Generative Models

In Signal Processing

Citations in the corpus

High-Resolution Image Synthesis with Latent Diffusion Models (2021, background): “results in image synthesis [29,82] and beyond [7,44,47,56], and define the state-of-the-art in class-conditional image synthesis [15,30] and super-resolution [70].”
Classifier-Free Diffusion Guidance (2022, methodology): “The reverse process mean comes from an estimate xθ(zλ) ≈ x plugged into q(zλ′ |zλ,x) [5, 8] (xθ also receives λ as input, but we suppress this to keep our notation clean).”
Score-based Generative Modeling in Latent Space (2021, background): “Additionally, [71, 4, 25] concurrently with us proposed likelihood-based training of SGMs in data space5.”
Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Models (2022, methodology): “Besides, we can also extend Theorem 1 to DPMs with continuous timesteps (Song et al., 2020b; Kingma et al., 2021), where their corresponding optimal mean and variance are also determined by the score function in an analytic form (see Appendix E.1 for the extension).”
Diffusion Autoencoders: Toward a Meaningful and Decodable Representation (2021, methodology): “Recent studies have improved DPMs further in both theory and practice [25,29,31].”
Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding (2022, methodology): “[35] Diederik P Kingma, Tim Salimans, Ben Poole, and Jonathan Ho.”
Scalable Adaptive Computation for Iterative Generation (2022, methodology): “Similar to (Kingma et al., 2021; Chen et al., 2022c), we use a continuous-time noise schedule function γ(t); by default we use a cosine schedule, as in previous work (Nichol & Dhariwal, 2021).”
End-to-end protein–ligand complex structure generation with diffusion-based generative models (2022, positive): “Methods In this section, we describe the formulation of the equivariant diffusion-based model used in our framework, which is based on Variational Diffusion Models [26] and E(3) Equivariant Diffusion Models [29].”
Fast Lossless Neural Compression with Integer-Only Discrete Flows (2022, background): “Currently, hierarchical VAEs attain theoretical code lengths comparable to NFs (Maaløe et al., 2019; Ho et al., 2019a) and DPMs achieve the best code lengths (Kingma et al., 2021) (2.49 bits/dim on CIFAR10, 3.72 bits/dim on ImageNet32 and 3.40 bits/dim on ImageNet64).”
DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps (2022, background): “Let q t ( x t ) denote the marginal distribution of x t , DPMs choose noise schedules to ensure that q T ( x T ) ≈ N ( x T | 0 , ˜ σ 2 I ) for some ˜ σ > 0 , and the signal-to-noise-ratio (SNR) α 2 t /σ 2 t is strictly decreasing w.r.t. t [10].”
Flow Matching for Generative Modeling (2022, neutral): “However, (Nichol & Dhariwal, 2021) and (Kingma et al., 2021) only consider a restricted setting of Gaussian conditional paths defined by simple diffusion processes with a single parameter—in particular, it does not include our conditional OT path.”
Palette: Image-to-Image Diffusion Models (2021, background): “…models [Ho et al. 2020; Sohl-Dickstein et al. 2015; Song and Ermon 2020] have received a surge of recent interest [Austin et al. 2021; Cai et al. 2020; Hoogeboom et al. 2021; Kingma et al. 2021; Song et al. 2021; Vahdat et al. 2021], resulting in several key advances in modeling continuous data.”
Video Diffusion Models (2022, background): “This corresponds to a particular weighting w ( λ t ) for learning a scaled score estimate (cid:15) θ ( z t ) ≈ − σ t ∇ z t log p ( z t ) , where p ( z t ) is the true density of z t under x ∼ p ( x ) [22, 28, 48].”
A Variational Perspective on Diffusion-Based Generative Models and Score Matching (2021, background): “This has been explored in a concurrent work by Kingma et al. (2021) in a different framework.”
Diffusion probabilistic modeling of protein backbones in 3D for the motif-scaffolding problem (2022, positive): “[24] Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho.”
DiGress: Discrete Denoising diffusion for graph generation (2022, background): “…this diffusion process adds noise independently on each node and each edge, but this time the noise considered is Gaussian: ) This process can equivalently be written: where The variance is chosen as ( σ t ) 2 = 1 − ( α t ) 2 in order to obtain a variance-preserving process (Kingma et al., 2021).”
QARV: Quantization-Aware ResNet VAE for Lossy Image Compression (2023, background): “For example, recent studies have shown that many powerful generative models, such as auto-regressive models [1], Variational Autoencoders [2], Normalizing Flows [3], and diffusion probabilistic models [4], can also serve as highly effective lossless compressors.”
Continuous diffusion for categorical data (2022, methodology): “The impact of different weightings has been studied extensively for diffusion models of images (Karras et al., 2022; Kingma et al., 2021; Nichol and Dhariwal, 2021; Song et al., 2021a).”