Palette: Image-to-Image Diffusion Models

By Chitwan Saharia, William Chan, Huiwen Chang, Chris A. Lee, Chris Lee, Jonathan Ho, Tim Salimans, David J. Fleet, Mohammad Norouzi. 2021

2,308 citations recorded by the source corpus.

This paper develops a unified framework for image-to-image translation based on conditional diffusion models and evaluates this framework on four challenging image-to-image translation tasks, namely colorization, inpainting, uncropping, and JPEG restoration. Our simple implementation of image-to-image diffusion models outperforms strong GAN and regression baselines on all tasks, without task-specific hyper-parameter tuning, architecture customization, or any auxiliary loss or sophisticated new techniques needed. We uncover the impact of an L2 vs. L1 loss in the denoising diffusion objective on sample diversity, and demonstrate the importance of self-attention in the neural architecture through empirical studies. Importantly, we advocate a unified evaluation protocol based on ImageNet, with human evaluation and sample quality scores (FID, Inception Score, Classification Accuracy of a pre-trained ResNet-50, and Perceptual Distance against original images). We expect this standardized evaluation protocol to play a role in advancing image-to-image translation research. Finally, we show that a generalist, multi-task diffusion model performs as well or better than task-specific specialist counterparts. Check out https://diffusion-palette.github.io/ for an overview of the results and code.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models (2021, methodology): “To achieve better results, we explicitly ﬁne-tune our model to perform inpainting, similar to Saharia et al. (2021a).”
Image Super-Resolution via Iterative Refinement (2021, background): “However, subsequent work shows that L2 tends to generate greater diversity in SR3 outputs [34].”
GLIGEN: Open-Set Grounded Text-to-Image Generation (2023, background): “While conditional diffusion models [9, 47, 49] and GANs [24, 33, 42, 64] that take in input modalities other than text for inpainting, layout2img generation, etc.”
InstructPix2Pix: Learning to Follow Image Editing Instructions (2022, background): “Diffusion-based generative models Recent advances in diffusion models [59] have enabled state-of-the-art image synthesis [10, 18, 19, 53, 55, 60] as well as generative models of other modalities such as video [21, 58], audio [30], text [35] and network parameters [45].”
Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding (2022, methodology): “Diffusion models have seen wide success in image generation [28, 42, 62, 16, 29, 61], outperforming GANs in fidelity and diversity, without training instability and mode collapse issues [6, 16, 29].”
Elucidating the Design Space of Diffusion-Based Generative Models (2022, background): “Diffusion-based generative models [45] have emerged as a powerful new framework for neural image synthesis, in both unconditional [16, 36, 48] and conditional [17, 35, 36, 38, 39, 41, 42, 48] settings, even surpassing the quality of GANs [13] in certain situations [9].”
Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale (2023, methodology): “Diversity and quality: Fréchet Inception Score (FID) [17] is widely adopted for image generation evaluations, which captures the similarity between generated and real images at the distribution level in some feature space.”
Video Diffusion Models (2022, background): “Diffusion models have recently been producing high quality results in image generation and audio generation [e.g. 28, 39, 40, 16, 23, 36, 48, 60, 42, 10, 29], and there is signiﬁcant interest in validating diffusion models in new data modalities.”
Diffusion probabilistic modeling of protein backbones in 3D for the motif-scaffolding problem (2022, positive): “In our case, we condition on the motif structure, a task analogous to inpainting [29].”
eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers (2022, background): “Diffusion models can be directly applied to various inverse problems, such as superresolution [13, 64], inpainting [11, 47], deblurring [35, 82], and JPEG restoration [36, 62].”
ResShift: Efficient Diffusion Model for Image Super-resolution by Residual Shifting (2023, background): “Furthermore, it has also demonstrated great potential in solving several downstream low-level vision tasks, including image editing [4, 5], image inpainting [6, 7], image colorization [8, 9].”
Human Motion Diffusion Model (2022, methodology): “Local editing of images is typically deﬁned as an inpainting problem, where a part of the image is constant, and the inpainted part is denoised by the model, possibly under some condition (Song et al., 2020b; Saharia et al., 2022a).”
A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence (2023, background): “Diffusion probabilistic models (diffusion models) [55] and denoising diffusion probabilistic models [23, 38] have demonstrated state-of-the-art performance for image generation task [13, 37, 48, 50, 51, 57].”
Align Your Latents: High-Resolution Video Synthesis with Latent Diffusion Models (2023, methodology): “They have also been successfully used for various image editing and processing tasks [18,25,41,43,48,53,66,67,69,74,84].”
SegDiff: Image Segmentation with Diffusion Probabilistic Models (2021, methodology): “Con-currently with our work, diffusion models were applied to image-to-image translation tasks [40].”
BlobGAN: Spatially Disentangled Scene Representations (2022, methodology): “Other setups include using perceptual losses [14], implicit likelihood estimation [43], and more recently, diffusion models [49, 72].”
MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation (2022, background): “It has shown that DPMs can perform well in several image generation tasks, such as image inpainting [28], super-resolution [32, 38, 50], image restoration [17], image-to-image translation [36], etc.”
Broadly applicable and accurate protein design by integrating structure prediction networks and diffusion generative models (2022, neutral): “In the context of RF diffusion , inpainting training corresponds to learning the reverse process of a conditional generation task 53 .”