D2C: Diffusion-Decoding Models for Few-Shot Conditional Generation

By Abhishek Sinha, Jiaming Song, Chenlin Meng, Stefano Ermon. 2021

146 citations recorded by the source corpus.

Conditional generative models of high-dimensional images have many applications, but supervision signals from conditions to images can be expensive to acquire. This paper describes Diffusion-Decoding models with Contrastive representations (D2C), a paradigm for training unconditional variational autoencoders (VAEs) for few-shot conditional image generation. D2C uses a learned diffusion-based prior over the latent representations to improve generation and contrastive self-supervised learning to improve representation quality. D2C can adapt to novel generation tasks conditioned on labels or manipulation constraints, by learning from as few as 100 labeled examples. On conditional generation from new labels, D2C achieves superior performance over state-of-the-art VAEs and diffusion models. On conditional image manipulation, D2C generations are two orders of magnitude faster to produce over StyleGAN2 ones and are preferred by 50% - 60% of the human evaluators in a double-blind study.

Subfields: Artificial Intelligence, Cancer Research, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cancer Research

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

High-Resolution Image Synthesis with Latent Diffusion Models (2021, methodology): “[80] Abhishek Sinha, Jiaming Song, Chenlin Meng, and Stefano Ermon.”
Diffusion Autoencoders: Toward a Meaningful and Decodable Representation (2021, methodology): “We also compare qualitatively to D2C [45], which uses NVAE [50] decoder to perform a similar task in our Appendix H.”
Palette: Image-to-Image Diffusion Models (2021, background): “…Nichol 2021; Ho et al. 2020, 2021], audio synthesis [Chen et al. 2021a; Kong et al. 2020], and image super-resolution [Kadkhodaie and Simoncelli 2021; Saharia et al. 2021], as well as unpaired imageto-image translation [Sasaki et al. 2021] and image editing [Meng et al. 2021; Sinha et al. 2021].”
SDFusion: Multimodal 3D Shape Completion, Reconstruction, and Generation (2022, methodology): “In particular, diffusion models have shown impressive quality, diversity, and expressiveness in various tasks such as image synthesis [13,15,16,27], super-resolution [36], image editing [10, 22, 40], text-toimage synthesis [4, 19, 26, 33, 35], etc.”
eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers (2022, background): “For computational efficiency, a diffusion model is often trained on low-resolution images or latent variables, which are then transformed into high-resolution images by super-resolution diffusion models [27] or latent-to-image decoders [68, 77].”
Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow (2022, unknown): “[68, 89, 81] learns a denoising diffusion model in the latent spaces and combines it with variational auto-encoders.”
SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagery (2022, background): “…image datasets [1, 2, 3, 4, 5, 6, 7, 8], and have demonstrated strong performance on downstream tasks including image classiﬁcation [3, 4, 9, 10], image segmentation [3, 11], representation learning [12, 13, 14], image compression [12, 15], image reconstruction [1], and image generation [16].”
On Distillation of Guided Diffusion Models (2022, methodology): “Latent diffusion models (LDMs) [21,24,28,35] increase the training and inference efficiency of diffusion models (directly learned on the pixel-space) by modeling images in the latent space of a pre-trained regularized autoencoder, where the latent representations are usually of lower dimensionality ”
AudioLDM: Text-to-Audio Generation with Latent Diffusion Models (2023, unknown): “One of the solutions is to employ diffusion models in a small latent space, an approach used, for example, in image generation (Vahdat et al., 2021; Sinha et al., 2021; Rombach et al., 2022).”
Diffusion models as plug-and-play priors (2022, methodology): “Deep generative models, such as denoising diffusion probabilistic models [DDPMs; 37, 13] can capture the details of very complex distributions over high-dimensional continuous data p(x) [28, 7, 1, 36, 40, 15].”
Score-Based Generative Modeling with Critically-Damped Langevin Diffusion (2021, background): “, 2021), image editing (Meng et al., 2021; Sinha et al., 2021; Furusawa et al., 2021), super-resolution (Saharia et al.”
Diffusion Models in Vision: A Survey (2022, background): “[16] presents the diffusiondecoding model with contrastive representations (D2C), a”