Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding
By Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, Seyedeh Sara Mahdavi, Raphael Gontijo Lopes and 4 more. 2022
8,628 citations recorded by the source corpus.
We present Imagen, a text-to-image diffusion model with an unprecedented degree of photorealism and a deep level of language understanding. Imagen builds on the power of large transformer language models in understanding text and hinges on the strength of diffusion models in high-fidelity image generation. Our key discovery is that generic large language models (e.g. T5), pretrained on text-only corpora, are surprisingly effective at encoding text for image synthesis: increasing the size of the language model in Imagen boosts both sample fidelity and image-text alignment much more than increasing the size of the image diffusion model. Imagen achieves a new state-of-the-art FID score of 7.27 on the COCO dataset, without ever training on COCO, and human raters find Imagen samples to be on par with the COCO data itself in image-text alignment. To assess text-to-image models in greater depth, we introduce DrawBench, a comprehensive and challenging benchmark for text-to-image models. With DrawBench, we compare Imagen with recent methods including VQ-GAN+CLIP, Latent Diffusion Models, and DALL-E 2, and find that human raters prefer Imagen over other models in side-by-side comparisons, both in terms of sample quality and image-text alignment. See https://imagen.research.google/ for an overview of the results.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, General Social Sciences, Generative Models, Multimodal Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Yoshua Bengio
- Sergey Levine
- Quoc V. Le
- Percy Liang
- Luke S. Zettlemoyer
- Ruslan Salakhutdinov
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Ming-Hsuan Yang
- Trevor Darrell
- Xiaogang Wang
- Jitendra Malik
- Andrew Zisserman
- Andrea Vedaldi
In General Social Sciences
- Ruslan Salakhutdinov
- Geoffrey E. Hinton
- Zhe Gan
- Qinsheng Zhang
- Alexei A. Efros
- Aleksander Holynski
- Yilun Du
- Yoshua Bengio
In Generative Models
- Jun-Yan Zhu
- Eli Shechtman
- Richard Zhang
- Arash Vahdat
- Stefano Ermon
- Daniel Cohen-Or
- Alexei A. Efros
- Karsten Kreis
In Multimodal Learning
Citations in the corpus
- Evaluating the Social Impact of Generative AI Systems in Systems and Society (2023, background): “They include but are not limited to generative models such as text-based language models (BLOOM [30], GPT-3 [40], OPT[271]), text-to-image models (ImaGen [205], DALL·E[170], Stable Diffusion [200]), and increasingly multimodal models [53] (GPT-4 [171]).”
- Visual Instruction Tuning (2023, background): “…foundation vision models [27, 16], with strong capabilities in open-world visual understanding such as classification [40, 21, 57, 54, 39], detection [29, 62, 33], segmentation [25, 63, 58] and captioning [50, 28], as well as visual generation and editing [42, 43, 56, 15, 44, 30].”
- Scalable Diffusion Models with Transformers (2022, background): “Diffusion [19, 51] and score-based generative models [22, 53] have been particularly successful as generative models of images [32,43,45,47], in many cases outperforming generative adversarial networks (GANs) [12] which had previously been state-of-the-art.”
- GLIGEN: Open-Set Grounded Text-to-Image Generation (2023, methodology): “Text conditional autoregressive [52, 74] and diffusion [51, 56] models have demonstrated astonishing image quality and concept coverage, due to their more stable learning objectives and large-scale training on web image-text paired data.”
- InstructPix2Pix: Learning to Follow Image Editing Instructions (2022, background): “Recent text-to-image diffusion models [41,48,51,54] have shown to generate realistic images from arbitrary text captions.”
- LAION-5B: An open large-scale dataset for training next generation image-text models (2022, background): “Another recent success of multimodal learning is in image generation, where DALL-E [59] and later models [52, 60, 64, 66, 90] demonstrated the potential of text-guided image generation by producing high-quality images specific to the provided text.”
- Scaling Autoregressive Models for Content-Rich Text-to-Image Generation (2022, methodology): “The most recent impressive results have been attained with diffusion models [11, 12, 13], where the models are learned to condition on the text encoder of the CLIP [77] image-text model, or frozen text encoder (like T5 [35]) pretrained by language self-supervision.”
- The Gradient of Generative AI Release: Methods and Considerations (2023, methodology): “Some publicly-known systems, such as Google’s Imagen [83] and DeepMind’s Gopher [73], are examples.”
- Learning to Learn with Generative Models of Neural Network Checkpoints (2022, methodology): “Diffusion models (Sohl-Dickstein et al., 2015) have recently been shown to be highly effective for images (Ho et al., 2020; Nichol & Dhariwal, 2021; Dhariwal & Nichol, 2021; Ho et al., 2021; Ramesh et al., 2022; Saharia et al., 2022; Song & Ermon, 2019).”
- MusicLM: Generating Music From Text (2023, background): “In contrast, and inspired by progress in text-to-image generation (Ramesh et al., 2021; 2022; Saharia et al., 2022; Yu et al., 2022), recent work has explored generating audio from sequence-wide, high-level captions (Yang et al., 2022; Kreuk et al., 2022) such as “whistling with wind blowing” .”
- Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models (2023, methodology): “Saharia et al. (2022) and Kreuk et al. (2022) utilize language models (e.g., BERT (Devlin et al., 2018), T5 (Raffel et al., 2020)) for text-guided generation.”
- Restoring Vision in Adverse Weather Conditions With Patch-Based Denoising Diffusion Models (2022, background): “Denoising diffusion models have recently demonstrated remarkable success in various generative modeling tasks [17], [18], [19], [20].”
- Align Your Latents: High-Resolution Video Synthesis with Latent Diffusion Models (2023, methodology): “We use noise augmentation with noise level conditioning [29, 68] and train the super resolution (SR) model gθ,φ (on images or latents) via”
- Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models (2022, methodology): “Many DMs have reacted to the concerns raised on large-scale training data by either not releasing the model [31], only deploying it in a controlled environment with dedicated guardrails in place [29] or rigorously filtering the training data of the published model [25].”
- T2I-CompBench: A Comprehensive Benchmark for Open-world Compositional Text-to-image Generation (2023, neutral): “DrawBench [3] consists of 200 prompts to evaluate counting, compositions, conflicting, and writing skills.”
- A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence (2023, background): “Diffusion probabilistic models (diffusion models) [55] and denoising diffusion probabilistic models [23, 38] have demonstrated state-of-the-art performance for image generation task [13, 37, 48, 50, 51, 57].”