Improved Denoising Diffusion Probabilistic Models
By Alex Nichol, Prafulla Dhariwal. 2021
5,632 citations recorded by the source corpus.
Denoising diffusion probabilistic models (DDPM) are a class of generative models which have recently been shown to produce excellent samples. We show that with a few simple modifications, DDPMs can also achieve competitive log-likelihoods while maintaining high sample quality. Additionally, we find that learning variances of the reverse diffusion process allows sampling with an order of magnitude fewer forward passes with a negligible difference in sample quality, which is important for the practical deployment of these models. We additionally use precision and recall to compare how well DDPMs and GANs cover the target distribution. Finally, we show that the sample quality and likelihood of these models scale smoothly with model capacity and training compute, making them easily scalable. We release our code at https://github.com/openai/improved-diffusion
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models, Signal Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Sergey Levine
- Mohit Bansal
- Jianfeng Gao
- Quoc V. Le
- Percy Liang
- Ruslan Salakhutdinov
- Luke S. Zettlemoyer
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Ming-Hsuan Yang
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Jitendra Malik
- Chen Change Loy
- Andrew Zisserman
In Generative Models
- Arash Vahdat
- Stefano Ermon
- Karsten Kreis
- Jiaming Song
- Jun-Yan Zhu
- Richard Zhang
- Jonathan Ho
- Yang Song
In Signal Processing
Citations in the corpus
- Classifier-Free Diffusion Guidance (2022, positive): “We use a p(λ) inspired by the cosine noise schedule of [10]: sampling λ is given by λ = −2 log tan(au+ b) for uniformly distributed u ∈ [0, 1], where b = arctan(e−λmax/2) and a = arctan(e−λmin/2)− b.”
- GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models (2021, methodology): “To achieve photorealism in the class-conditional setting, Dhariwal & Nichol (2021) augmented diffusion models with classifier guidance , a technique which allows diffusion models to condition on a classifier’s labels.”
- Generative Adversarial Networks (2021, neutral): “It is worth mentioning also normalizing flows [51] and score based generative models [67, 66, 48], which become very popular recently and show competitive performance with GANs.”
- Variational Diffusion Models (2021, background): “Diffusion models have recently shown impressive results in image [Ho et al., 2020, Song et al., 2021b, Nichol and Dhariwal, 2021] and audio generation [Kong et al., 2020, Chen et al., 2020] in terms of perceptual quality, but have yet to match autoregressive models on density estimation benchmarks.”
- Score-based Generative Modeling in Latent Space (2021, background): “Further, [26] propose empirical variance reduction techniques for discretized diffusions and introduce a new, heuristically motivated, noise schedule.”
- Cascaded Diffusion Models for High Fidelity Image Generation (2021, methodology): “Conditional diffusion models In the conditional generation setting, the data x0 has an associated conditioning signal c, for example a label in the case of class-conditional generation, or a low resolution image in the case of super-resolution [21, 16].”
- Diffusion Models Beat GANs on Image Synthesis (2021, methodology): “Nichol and Dhariwal [49] find that fixing”
- Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Models (2022, methodology): “Following Nichol & Dhariwal (2021), we train 500K iterations with a batch size of 128, use a learning rate of 0.0001 with the AdamW optimizer (Loshchilov & Hutter, 2017) and use an exponential moving average (EMA) with a rate of 0.9999.”
- High Fidelity Visualization of What Your Self-Supervised Representation Knows About (2021, background): “…considered other conditional generative approaches (we discussed some of the alternatives in section 2), the choice of the reverse diffusion approach (Sohl-Dickstein et al., 2015; Ho et al., 2020; Song & Ermon, 2019; 2020; Song et al., 2021a;b; Nichol & Dhariwal, 2021) is not entirely arbitrary.”
- Diffusion Autoencoders: Toward a Meaningful and Decodable Representation (2021, methodology): “Recent attempts to improve sampling speed focus on finding a better sampling interval or noise schedule [25, 29, 31, 36], or using more efficient solvers to solve the score-based ODE counterpart [25].”
- Vector Quantized Diffusion Model for Text-to-Image Synthesis (2021, background): “We assess the performance of our model in terms of FID and compare with a variety of previously established models [3,12,15,16,35].”
- Scalable Diffusion Models with Transformers (2022, methodology): “Nichol and Dhariwal’s seminal work improving diffusion models [36, 9] is most related to us—there, they analyzed the scalability properties of the U-Net architecture class.”
- Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding (2022, methodology): “Diffusion models have seen wide success in image generation [28, 42, 62, 16, 29, 61], outperforming GANs in fidelity and diversity, without training instability and mode collapse issues [6, 16, 29].”
- Scalable Adaptive Computation for Iterative Generation (2022, methodology): “By default we use a cosine schedule, as in previous work (Nichol & Dhariwal, 2021) but find it is some-times unstable for higher resolution images.”
- Learning to Learn with Generative Models of Neural Network Checkpoints (2022, methodology): “Given an input corrupted with noise, diffusion models can be parameterized to predict either the signal or the noise (Ho et al., 2020; Nichol & Dhariwal, 2021).”