eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers

By Yogesh Kumar Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Qinsheng Zhang, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine and 3 more. 2022

1,087 citations recorded by the source corpus.

Large-scale diffusion-based generative models have led to breakthroughs in text-conditioned high-resolution image synthesis. Starting from random noise, such text-to-image diffusion models gradually synthesize images in an iterative fashion while conditioning on text prompts. We find that their synthesis behavior qualitatively changes throughout this process: Early in sampling, generation strongly relies on the text prompt to generate text-aligned content, while later, the text conditioning is almost entirely ignored. This suggests that sharing model parameters throughout the entire generation process may not be ideal. Therefore, in contrast to existing works, we propose to train an ensemble of text-to-image diffusion models specialized for different synthesis stages. To maintain training efficiency, we initially train a single model, which is then split into specialized models that are trained for the specific stages of the iterative generation process. Our ensemble of diffusion models, called eDiff-I, results in improved text alignment while maintaining the same inference computation cost and preserving high visual quality, outperforming previous large-scale text-to-image diffusion models on the standard benchmark. In addition, we train our model to exploit a variety of embeddings for conditioning, including the T5 text, CLIP text, and CLIP image embeddings. We show that these different embeddings lead to different behaviors. Notably, the CLIP image embedding allows an intuitive way of transferring the style of a reference image to the target text-to-image output. Lastly, we show a technique that enables eDiff-I's"paint-with-words"capability. A user can select the word in the input text and paint it in a canvas to control the output, which is very handy for crafting the desired image in mind. The project page is available at https://deepimagination.cc/eDiff-I/

Subfields: Computer Vision and Pattern Recognition, Generative Models, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

In Multimodal Learning

Citations in the corpus

Point-E: A System for Generating 3D Point Clouds from Complex Prompts (2022, background): “…is now possible to generate and modify high-quality images from natural language descriptions in a number of seconds (Ramesh et al., 2021; Ding et al., 2021; Nichol et al., 2021; Ramesh et al., 2022; Gafni et al., 2022; Yu et al., 2022; Saharia et al., 2022; Feng et al., 2022; Balaji et al., 2022).”
Hunyuan-DiT: A Powerful Multi-Resolution Diffusion Transformer with Fine-Grained Chinese Understanding (2024, methodology): “In contrast, eDiff-I [3] and Swinv2-Imagen [18] fuse the two encoders, CLIP and T5, to further improve their text understanding capabilities.”
Champ: Controllable and Consistent Human Image Animation with 3D Parametric Guidance (2024, background): “Diffusion-based models [3,17,27, 29, 32,34] have rapidly emerged as a fundamental component in the domain of text-to-image generation, renowned for their capacity to yield highly promising generative outcomes.”
Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generation (2023, positive): “As pointed out by (Balaji et al., 2022), various diffusion steps assume distinct roles during the generation process.”
Text-to-3D using Gaussian Splatting (2023, methodology): “To generate high-resolution images, current solutions either adopt a cascaded structure that consists of a low-resolution diffusion model and several super-resolution models (Saharia et al., 2022; Balaji et al., 2022; Alex et al., 2023) or trains the diffusion model in latent space with an autoencod”
Synthetic Data from Diffusion Models Improves ImageNet Classification (2023, methodology): “Several text-to-image models including Stable Diffusion [47], DALL-E 2 [44], Imagen [50], eDiff [1], and GLIDE [40] have produced evocative high-resolution images.”
Efficient Diffusion Training via Min-SNR Weighting Strategy (2023, methodology): “Finally, some works [14, 1] have proposed using the Mixture of Experts (MoE) approach to han-7442 dle noise from different levels, which can boost the performance of diffusion models, but require a larger number of parameters and longer training time.”
VideoComposer: Compositional Video Synthesis with Motion Controllability (2023, background): “To further transfer the style from one image to the synthesized video, we choose the image embedding as the stylistic guidance, following [4, 26].”
Exploiting Diffusion Prior for Real-World Image Super-Resolution (2023, neutral): “2.”
LaVie: High-Quality Video Generation with Cascaded Latent Diffusion Models (2023, neutral): “The Bund, Shanghai, with the ship moving on the river, oil painting.”
ControlVideo: Training-free Controllable Text-to-Video Generation (2023, background): “Through pre-training on billions of image-text pairs, large-scale generative models [1, 2, 3, 4, 14, 22, 26, 27, 29, 32, 33, 42] have made remarkable progress in creative and photo-realistic image generation.”
VideoCrafter1: Open Diffusion Models for High-Quality Video Generation (2023, background): “Diffusion models (DMs) [26, 46, 48] have recently shown unprecedented capability in content generation field, especially in text-to-image (T2I) generation [9, 21, 25, 38, 39, 41–43 , 62].”
Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models (2023, background): “Recent advancements in text-based image generation [5, 17, 37, 39, 42] have demonstrated an unprecedented ability to generate diverse and creative imagery provided a freeform text prompt.”
Adversarial Diffusion Distillation (2023, background): “While diffusion models achieve remarkable performance in synthesizing and editing high-resolution images [3, 53, 54] and videos [4, 21], their iterative nature hinders real-time application.”
Consistency Models (2023, positive): “(5) bears strong resemblance to many successful diffusion models (Karras et al., 2022; Balaji et al., 2022), making it easier to borrow powerful diffusion model architectures for constructing consistency models.”
Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models (2023, methodology): “We also examine the idea of ensemble denoiser [2] by finetuning the level-1 experts of each model.”
StyleGAN-T: Unlocking the Power of GANs for Fast Large-Scale Text-to-Image Synthesis (2023, methodology): “We use zero-shot MS COCO to compare the performance of our model to the state-of-the-art quantitatively at 64 × 64 pixel output resolution in (Balaji et al., 2022; Lambda Labs, 2022); the other numbers are obtained from the respective papers or through correspondence with the authors. ing phase,…”
Structure and Content-Guided Video Synthesis with Diffusion Models (2023, methodology): “( Content Representation We utilize CLIP [32] to infer a content representation from both text inputs t and video inputs x similar to previous works [35, 3].”