Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding

By Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, Seyedeh Sara Mahdavi, Raphael Gontijo Lopes and 4 more. 2022

8,628 citations recorded by the source corpus.

We present Imagen, a text-to-image diffusion model with an unprecedented degree of photorealism and a deep level of language understanding. Imagen builds on the power of large transformer language models in understanding text and hinges on the strength of diffusion models in high-fidelity image generation. Our key discovery is that generic large language models (e.g. T5), pretrained on text-only corpora, are surprisingly effective at encoding text for image synthesis: increasing the size of the language model in Imagen boosts both sample fidelity and image-text alignment much more than increasing the size of the image diffusion model. Imagen achieves a new state-of-the-art FID score of 7.27 on the COCO dataset, without ever training on COCO, and human raters find Imagen samples to be on par with the COCO data itself in image-text alignment. To assess text-to-image models in greater depth, we introduce DrawBench, a comprehensive and challenging benchmark for text-to-image models. With DrawBench, we compare Imagen with recent methods including VQ-GAN+CLIP, Latent Diffusion Models, and DALL-E 2, and find that human raters prefer Imagen over other models in side-by-side comparisons, both in terms of sample quality and image-text alignment. See https://imagen.research.google/ for an overview of the results.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, General Social Sciences, Generative Models, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Jianfeng Gao
  3. Yoshua Bengio
  4. Sergey Levine
  5. Quoc V. Le
  6. Percy Liang
  7. Luke S. Zettlemoyer
  8. Ruslan Salakhutdinov

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Ming-Hsuan Yang
  4. Trevor Darrell
  5. Xiaogang Wang
  6. Jitendra Malik
  7. Andrew Zisserman
  8. Andrea Vedaldi

In General Social Sciences

  1. Ruslan Salakhutdinov
  2. Geoffrey E. Hinton
  3. Zhe Gan
  4. Qinsheng Zhang
  5. Alexei A. Efros
  6. Aleksander Holynski
  7. Yilun Du
  8. Yoshua Bengio

In Generative Models

  1. Jun-Yan Zhu
  2. Eli Shechtman
  3. Richard Zhang
  4. Arash Vahdat
  5. Stefano Ermon
  6. Daniel Cohen-Or
  7. Alexei A. Efros
  8. Karsten Kreis

In Multimodal Learning

  1. Zhe Gan
  2. Jianfeng Gao
  3. Mohit Bansal
  4. Lijuan Wang
  5. Linjie Li
  6. Zicheng Liu
  7. Devi Parikh
  8. Jianfeng Wang

Citations in the corpus