Tackling the Generative Learning Trilemma with Denoising Diffusion GANs

By Zhisheng Xiao, Karsten Kreis, Arash Vahdat. 2021

766 citations recorded by the source corpus.

A wide variety of deep generative models has been developed in the past decade. Yet, these models often struggle with simultaneously addressing three key requirements including: high sample quality, mode coverage, and fast sampling. We call the challenge imposed by these requirements the generative learning trilemma, as the existing models often trade some of them for others. Particularly, denoising diffusion models have shown impressive sample quality and diversity, but their expensive sampling does not yet allow them to be applied in many real-world applications. In this paper, we argue that slow sampling in these models is fundamentally attributed to the Gaussian assumption in the denoising step which is justified only for small step sizes. To enable denoising with large steps, and hence, to reduce the total number of denoising steps, we propose to model the denoising distribution using a complex multimodal distribution. We introduce denoising diffusion generative adversarial networks (denoising diffusion GANs) that model each denoising step using a multimodal conditional GAN. Through extensive evaluations, we show that denoising diffusion GANs obtain sample quality and diversity competitive with original diffusion models while being 2000$\times$ faster on the CIFAR-10 dataset. Compared to traditional GANs, our model exhibits better mode coverage and sample diversity. To the best of our knowledge, denoising diffusion GAN is the first model that reduces sampling cost in diffusion models to an extent that allows them to be applied to real-world applications inexpensively. Project page and code can be found at https://nvlabs.github.io/denoising-diffusion-gan

Subfields: Computer Vision and Pattern Recognition, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps (2022, background): “For instance, defining a low-dimensional latent variable for DPMs [34]; designing special diffusion processes with bounded score functions [35]; combining GANs with the reverse process of DPMs [36].”
ProDiff: Progressive Fast Diffusion Model for High-Quality Text-to-Speech (2022, methodology): “It has been demonstrated that diffusion models [5, 52] can learn diverse data distribution in multiple domains, such as images and time series.”
Align Your Latents: High-Resolution Video Synthesis with Latent Diffusion Models (2023, background): “However, despite advances in model distillation [49,52,73] and accelerated sampling [4,11,12,36,45,46,80,96,100,108], DMs generally require repeated evaluations of a computationally demanding large U-Net.”
Diffusion models in medical imaging: A comprehensive survey (2022, background): “These requirements include (i) high-quality sampling, (ii) mode coverage and sample diversity, and (iii) fast execution time and computationally inexpensive sampling [26] (see Figure 3).”
Stochastic Interpolants: A Unifying Framework for Flows and Diffusions (2023, methodology): “The SDE framework, as it has been originally presented, introduces a number of complexities which are not a priori well motivated, including the dependence on mapping to a normal density and the complicated tuning of the time parameterization and noise scheduling [49, 18] as well as the underlying s”
Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow (2022, background): “It is likely to further improve rectiﬁed ﬂow with proper data augmentation techniques, or the combination of GANs such as those proposed by TDPM [99] and denoising diffusion GAN [91].”
Generating Novel, Designable, and Diverse Protein Structures by Equivariantly Diffusing Oriented Residue Clouds (2023, methodology): “Generative modeling trilemma Generative models generally contend with a trilemma in optimizing between quality (physicality and designability of protein structures), mode coverage (novelty and diversity of structures) and sampling time (Xiao et al., 2022).”
InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation (2023, methodology): “Inference Time FID-5k CLIP SD 1.4-DPM Solver (25 step)[5, 19] 0”
Denoising Diffusion Models for Plug-and-Play Image Restoration (2023, methodology): “It is worth noting that DDRM is structurally similar to our method, as both first predict x 0 and then add noise to forward sample x t − 1 .”
LION: Latent Point Diffusion Models for 3D Shape Generation (2022, background): “[62] Zhisheng Xiao, Karsten Kreis, and Arash Vahdat.”
Building Normalizing Flows with Stochastic Interpolants (2022, unknown): “Namely, the SDE must evolve for infinite time to connect the distributions, the parameterization of the time steps remains heuristic (Feller, 1949; Xiao et al., 2022), and the criticality of noise, as well as the score, is not absolutely apparent (Bansal et al.”
Adversarial Diffusion Distillation (2023, methodology): “For faster sampling, Denoising Diffusion GANs [70] are introduced as a method to enable sampling with few steps.”
DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models (2022, unknown): “In addition, combining DPMs with GANs (Xiao et al., 2022; Wang et al., 2022) improves the sample quality of GANs and sampling speed of DPMs.”
DDFM: Denoising Diffusion Model for Multi-Modality Image Fusion (2023, background): “Moreover, its generation process is interpretable, as it is based on denoising diffusion to generate images, enabling a better understanding of the image generation process [57].”
AnoDDPM: Anomaly Detection with Denoising Diffusion Probabilistic Models using Simplex Noise (2022, background): “Denoising diffusion probabilistic models (DDPMs) [7] are capable of generating samples from complex data distributions, with superior mode coverage [27] over generative adversarial networks (GANs) and variational autoen-coders (VAEs) [4] (a desirable property for anomaly detection and small…”