Cascaded Diffusion Models for High Fidelity Image Generation

By Jonathan Ho, Chitwan Saharia, William Chan, David J. Fleet, Mohammad Norouzi, Tim Salimans. 2021

1,672 citations recorded by the source corpus.

We show that cascaded diffusion models are capable of generating high fidelity images on the class-conditional ImageNet generation benchmark, without any assistance from auxiliary image classifiers to boost sample quality. A cascaded diffusion model comprises a pipeline of multiple diffusion models that generate images of increasing resolution, beginning with a standard diffusion model at the lowest resolution, followed by one or more super-resolution diffusion models that successively upsample the image and add higher resolution details. We find that the sample quality of a cascading pipeline relies crucially on conditioning augmentation, our proposed method of data augmentation of the lower resolution conditioning inputs to the super-resolution models. Our experiments show that conditioning augmentation prevents compounding error during sampling in a cascaded model, helping us to train cascading pipelines achieving FID scores of 1.48 at 64x64, 3.52 at 128x128 and 4.88 at 256x256 resolutions, outperforming BigGAN-deep, and classification accuracy scores of 63.02% (top-1) and 84.06% (top-5) at 256x256, outperforming VQ-VAE-2.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Hierarchical Text-Conditional Image Generation with CLIP Latents (2022, background): “Concurrently, diffusion models [46, 48, 25] have emerged as a promising generative modeling framework, pushing the state-of-the-art on image and video generation tasks [11, 26, 24].”
Classifier-Free Diffusion Guidance (2022, methodology): “These results compare favorably to [3, 6] and are currently state-of-the-art for this data set as far as we are aware for models that use T ≈ 256 steps (the ADM result uses 250 steps, and the CDM result is a two-stage model with 4000 steps each).”
Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors (2022, methodology): “Other likelihood-based models include autoregressive models [58, 43, 13, 8] and diffusion models [11, 21, 20].”
GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models (2021, background): “Ho & Salimans (2021) achieved similar results without a separately trained classiﬁer through the use of classiﬁer-free guidance , a form of guidance that interpolates between predictions from a diffusion model with and without labels.”
Image Super-Resolution via Iterative Refinement (2021, neutral): “models trained directly, which has been confirmed in more recent work [72].”
Diffusion Autoencoders: Toward a Meaningful and Decodable Representation (2021, background): “Unlike in other conditional DPMs [23, 32, 45] that use spatial conditional variables (e.”
Vector Quantized Diffusion Model for Text-to-Image Synthesis (2021, background): “Diffusion generative models were first proposed in [55] and achieved strong results on image generation [12,20,21,35] and image super super-resolution [48] recently.”
Scalable Diffusion Models with Transformers (2022, background): “Improvements in DDPMs over the past two years have largely been driven by improved sampling techniques [19, 25, 52], most notably classifierfree guidance [21], reformulating diffusion models to predict noise instead of pixels [19] and using cascaded DDPM pipelines where low-resolution base diffusion”
InstructPix2Pix: Learning to Follow Image Editing Instructions (2022, background): “Diffusion-based generative models Recent advances in diffusion models [59] have enabled state-of-the-art image synthesis [10, 18, 19, 53, 55, 60] as well as generative models of other modalities such as video [21, 58], audio [30], text [35] and network parameters [45].”
Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding (2022, methodology): “To pursue this research direction further, we introduce Imagen, a text-to-image diffusion model that combines the power of transformer language models (LMs) [15, 54] with high-fidelity diffusion models [28, 29, 16, 43] to deliver an unprecedented degree of photorealism and a deep level of language u”
Scalable Adaptive Computation for Iterative Generation (2022, methodology): “Common diffusion models for images and videos can be roughly divided into pixel diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020; Song et al., 2020; Dhariwal & Nichol, 2022; Ho et al., 2022a; Karras et al., 2022) and latent diffusion models (Rombach et al., 2022).”
Learning to Learn with Generative Models of Neural Network Checkpoints (2022, background): “, 2015) have recently been shown to be highly effective for images (Ho et al., 2020; Nichol & Dhariwal, 2021; Dhariwal & Nichol, 2021; Ho et al., 2021; Ramesh et al., 2022; Saharia et al., 2022; Song & Ermon, 2019).”
Elucidating the Design Space of Diffusion-Based Generative Models (2022, methodology): “This model achieved a new state-of-the-art FID of 1.36 compared to the previous record of 1.48 [17].”
MaskGIT: Masked Generative Image Transformer (2022, methodology): “In contrast, likelihood-based methods, such as Variational Autoencoders (VAEs) [29, 43], Diffusion Models [12, 24, 40] and Autoregressive Models [34, 44], offer distribution coverage and hence can generate more diverse samples [40,43, 44].”
Palette: Image-to-Image Diffusion Models (2021, background): “On the class-conditional ImageNet generation challenge they have outperformed strong GAN baselines in terms of FID scores [Dhariwal and Nichol 2021; Ho et al. 2021].”
Video Diffusion Models (2022, background): “1 INTRODUCTION Diffusion models have recently been producing high quality results in image generation and audio generation (e.g. Kingma et al., 2021; Saharia et al., 2021a;b; Dhariwal & Nichol, 2021; Ho et al., 2021; Nichol et al., 2021; Song et al., 2021b; Whang et al., 2021; Salimans & Ho, 2021; C”
SmartBrush: Text and Shape Guided Object Inpainting with Diffusion Model (2022, neutral): “, unconditional image generation [7, 8, 25, 26], text-toimage generation [18–21], video generation [6], image inpainting [1,2,14,16], image translation [15,27,30], and image editing [4, 5, 10].”