Diffusion Models Beat GANs on Image Synthesis
By Prafulla Dhariwal, Alex Nichol. 2021
12,477 citations recorded by the source corpus.
We show that diffusion models can achieve image sample quality superior to the current state-of-the-art generative models. We achieve this on unconditional image synthesis by finding a better architecture through a series of ablations. For conditional image synthesis, we further improve sample quality with classifier guidance: a simple, compute-efficient method for trading off diversity for fidelity using gradients from a classifier. We achieve an FID of 2.97 on ImageNet 128$\times$128, 4.59 on ImageNet 256$\times$256, and 7.72 on ImageNet 512$\times$512, and we match BigGAN-deep even with as few as 25 forward passes per sample, all while maintaining better coverage of the distribution. Finally, we find that classifier guidance combines well with upsampling diffusion models, further improving FID to 3.94 on ImageNet 256$\times$256 and 3.85 on ImageNet 512$\times$512. We release our code at https://github.com/openai/guided-diffusion
Subfields: Biophysics, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Generative Models
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Biophysics
- Yang Song
- Prafulla Dhariwal
- Richard Zhang
- Jun-Yan Zhu
- Jonathan Ho
- Diederik P. Kingma
- Jong Chul Ye
- Stefano Ermon
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Ming-Hsuan Yang
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Jitendra Malik
- Andrea Vedaldi
- Andrew Zisserman
In Statistical and Nonlinear Physics
- Arash Vahdat
- Yang Song
- Stefano Ermon
- Karsten Kreis
- Jonathan Ho
- Prafulla Dhariwal
- Diederik P. Kingma
- Jun Zhu
In Generative Models
- Arash Vahdat
- Jun-Yan Zhu
- Stefano Ermon
- Eli Shechtman
- Richard Zhang
- Karsten Kreis
- Alexei A. Efros
- Daniel Cohen-Or
Citations in the corpus
- Classifier-Free Diffusion Guidance (2022, methodology): “These results compare favorably to [3, 6] and are currently state-of-the-art for this data set as far as we are aware for models that use T ≈ 256 steps (the ADM result uses 250 steps, and the CDM result is a two-stage model with 4000 steps each).”
- Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors (2022, methodology): “Other likelihood-based models include autoregressive models [58, 43, 13, 8] and diffusion models [11, 21, 20].”
- GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models (2021, methodology): “To achieve photorealism in the class-conditional setting, Dhariwal & Nichol (2021) augmented diffusion models with classifier guidance , a technique which allows diffusion models to condition on a classifier’s labels.”
- Vector-quantized Image Modeling with Improved VQGAN (2021, neutral): “…Adversarial Networks (GANs) (Goodfellow et al., 2014; Zhang et al., 2019b), Variational Autoencoders (VAEs) (Kingma & Welling, 2014; Vahdat & Kautz, 2020), Diffusion Models (Song & Ermon, 2019; Dhariwal & Nichol, 2021) and Autoregressive Models (van den Oord et al., 2016; Parmar et al., 2018).”
- ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image Synthesis (2021, background): “[65], DDPM [26], ADM [13]) and also approaches the inference speed of TT [18], which only consists of a single autoregressive stage.”
- D2C: Diffusion-Decoding Models for Few-Shot Conditional Generation (2021, methodology): “Diffusion models [27] achieves high sample quality without adversarial training, but its latent dimension must be equal to the image dimension.”
- Score-based Generative Modeling in Latent Space (2021, methodology): “Variants of SGMs have been applied to images [1, 2, 5, 6], audio [7, 8, 9, 10], graphs [11] and point clouds [12, 13].”
- On Fast Sampling of Diffusion Probabilistic Models (2021, methodology): “Most recently, these models have obtained the state-of-the-art results in several important domains, including image synthesis (Ho et al., 2020; Song et al., 2020b; Dhariwal and Nichol, 2021), audio synthesis (Kong et al., 2020b; Chen et al., 2020), and 3-”
- Taming Transformers for High-Resolution Image Synthesis (2020, methodology): “12 (which replaces the previous qualitative comparison), and results on class-conditional ImageNet sampling from [49, 48, 15] in Tab.”
- Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Models (2022, background): “Recently, it is shown that DPMs are able to produce high-quality samples (Ho et al., 2020; Nichol & Dhariwal, 2021; Song et al., 2020b; Dhariwal & Nichol, 2021), which are comparable or even superior to the current state-of-the-art GAN models (Goodfellow et al., 2014; Brock et al., 2018; Wu et al.,…”
- High Fidelity Visualization of What Your Self-Supervised Representation Knows About (2021, positive): “In contrast with Dhariwal & Nichol (2021) we don’t use the input gradient of a classifier to bias the reversed diffusion process towards more probable images, nor do we use any label information for training our model – recall that our goal is building a visualization tool for SSL models that train…”
- Vector Quantized Diffusion Model for Text-to-Image Synthesis (2021, methodology): “We start with a vector quantized variational autoencoder (VQ-VAE) and model its latent space by learning a parametric model using a conditional variant of the Denoising Diffusion Probabilistic Model (DDPM) [23, 59], which has been applied to image synthesis with compelling results [12].”
- Scalable Diffusion Models with Transformers (2022, methodology): “For example, diffusion models have been at the forefront of recent advances in image generation [9, 46]; yet, they all adopt a convolutional U-Net architecture as the de-facto choice of backbone.”
- GLIGEN: Open-Set Grounded Text-to-Image Generation (2023, background): “While conditional diffusion models [10, 53, 55] and GANs [26, 37, 48, 71] that take in input modalities other than text for inpainting, layout2img generation, etc.”