Hierarchical Text-Conditional Image Generation with CLIP Latents

By Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, Mark Chen. arXiv.org, 2022

9,210 citations recorded by the source corpus.

Contrastive models like CLIP have been shown to learn robust representations of images that capture both semantics and style. To leverage these representations for image generation, we propose a two-stage model: a prior that generates a CLIP image embedding given a text caption, and a decoder that generates an image conditioned on the image embedding. We show that explicitly generating image representations improves image diversity with minimal loss in photorealism and caption similarity. Our decoders conditioned on image representations can also produce variations of an image that preserve both its semantics and style, while varying the non-essential details absent from the image representation. Moreover, the joint embedding space of CLIP enables language-guided image manipulations in a zero-shot fashion. We use diffusion models for the decoder and experiment with both autoregressive and diffusion models for the prior, finding that the latter are computationally more efficient and produce higher-quality samples.

Subfields: Computer Vision and Pattern Recognition, Generative Models, Multimodal Learning

View Paper: https://arxiv.org/abs/2204.06125

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Ming-Hsuan Yang
  4. Trevor Darrell
  5. Xiaogang Wang
  6. Jitendra Malik
  7. Andrew Zisserman
  8. Andrea Vedaldi

In Generative Models

  1. Jun-Yan Zhu
  2. Richard Zhang
  3. Eli Shechtman
  4. Arash Vahdat
  5. Stefano Ermon
  6. Alexei A. Efros
  7. Karsten Kreis
  8. Daniel Cohen-Or

In Multimodal Learning

  1. Mohit Bansal
  2. Zhe Gan
  3. Jianfeng Gao
  4. Devi Parikh
  5. Lijuan Wang
  6. Marcus Rohrbach
  7. Linjie Li
  8. Trevor Darrell

Citations in the corpus