Hierarchical Text-Conditional Image Generation with CLIP Latents
By Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, Mark Chen. arXiv.org, 2022
9,210 citations recorded by the source corpus.
Contrastive models like CLIP have been shown to learn robust representations of images that capture both semantics and style. To leverage these representations for image generation, we propose a two-stage model: a prior that generates a CLIP image embedding given a text caption, and a decoder that generates an image conditioned on the image embedding. We show that explicitly generating image representations improves image diversity with minimal loss in photorealism and caption similarity. Our decoders conditioned on image representations can also produce variations of an image that preserve both its semantics and style, while varying the non-essential details absent from the image representation. Moreover, the joint embedding space of CLIP enables language-guided image manipulations in a zero-shot fashion. We use diffusion models for the decoder and experiment with both autoregressive and diffusion models for the prior, finding that the latter are computationally more efficient and produce higher-quality samples.
Subfields: Computer Vision and Pattern Recognition, Generative Models, Multimodal Learning
View Paper: https://arxiv.org/abs/2204.06125
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Ming-Hsuan Yang
- Trevor Darrell
- Xiaogang Wang
- Jitendra Malik
- Andrew Zisserman
- Andrea Vedaldi
In Generative Models
- Jun-Yan Zhu
- Richard Zhang
- Eli Shechtman
- Arash Vahdat
- Stefano Ermon
- Alexei A. Efros
- Karsten Kreis
- Daniel Cohen-Or
In Multimodal Learning
Citations in the corpus
- Emergent Abilities of Large Language Models (2022, neutral): “Other frontier tasks could include prompting in multiple modalities (Alayrac et al., 2022; Ramesh et al., 2022).”
- Large Language Models Are Human-Level Prompt Engineers (2022, background): “…to communicate with machines and may be of great relevance not only to LLMs but to other generalist models such as prompted image synthesizers (Rombach et al., 2022; Ramesh et al., 2022), for which public interest in prompt design and generation has also emerged (see Appendix A for examples).”
- Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned (2022, background): “Although our work focuses on language models, there is prior art on how to red team multi-modal generative models [39] in the context of DALL•E 2 [44].”
- Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling (2023, background): “Beyond NLP, transformers have also made big splashes as generative models in areas as diverse as text-to-image synthesis (Ramesh et al., 2022; Crowson et al., 2022; Rombach et al., 2022), protein modeling (Jumper et al.”
- Flamingo: a Visual Language Model for Few-Shot Learning (2022, background): “– Dialogue 1: Available from DALL·E 2 [89].”
- Visual Instruction Tuning (2023, background): “…foundation vision models [27, 16], with strong capabilities in open-world visual understanding such as classification [40, 21, 57, 54, 39], detection [29, 62, 33], segmentation [25, 63, 58] and captioning [50, 28], as well as visual generation and editing [42, 43, 56, 15, 44, 30].”
- Scalable Diffusion Models with Transformers (2022, methodology): “Finally, transformers have been explored in DDPMs to synthesize non-spatial data; e.g., to generate CLIP image embeddings in DALL E 2 [46, 41].”
- Stable and low-precision training for large-scale vision-language models (2023, methodology): “We examine CLIP-style models because of their importance in computer vision: CLIP-style models reach state-ofthe-art performance on a wide range of image classification tasks [46, 65, 44, 7] and underlie image generation methods such as DALLE·2 [49] and Stable Diffusion [51].”
- InstructPix2Pix: Learning to Follow Image Editing Instructions (2022, background): “Recent text-to-image diffusion models [41,48,51,54] have shown to generate realistic images from arbitrary text captions.”
- LAION-5B: An open large-scale dataset for training next generation image-text models (2022, background): “Another recent success of multimodal learning is in image generation, where DALL-E [59] and later models [52, 60, 64, 66, 90] demonstrated the potential of text-guided image generation by producing high-quality images specific to the provided text.”
- Scaling Autoregressive Models for Content-Rich Text-to-Image Generation (2022, methodology): “Similar to DALL-E 2 [12], we use guidance scale 1.”
- Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding (2022, methodology): “Figure 3: Comparison between Imagen and DALL-E 2 [56], GLIDE [43], VQ-GAN+CLIP [12] and Latent Diffusion [59] on DrawBench: User preference rates (with 95% confidence intervals) for image-text alignment and image fidelity.”
- The Gradient of Generative AI Release: Methods and Considerations (2023, methodology): “This method can also transition to API or downloadable, as seen with OpenAI’s DALL·E 2 [77] was”
- Multiview Compressive Coding for 3D Reconstruction (2023, methodology): “We demonstrate the generality of MCC by experimenting on six diverse data sources: CO3D [50], Hypersim [51], Taskon-omy [81], ImageNet [7], in-the-wild iPhone captures and DALL · E 2 [47] generations.”
- Learning to Learn with Generative Models of Neural Network Checkpoints (2022, methodology): “We follow DALL · E 2’s (Ramesh et al., 2022) normalization scheme, where the data is scaled such that the variance of the marginal distribution matches the variance of ImageNet pixels scaled to [-1, 1], for which diffusion hyperparameters have been tuned.”
- Elucidating the Design Space of Diffusion-Based Generative Models (2022, background): “Diffusion-based generative models [45] have emerged as a powerful new framework for neural image synthesis, in both unconditional [16, 36, 48] and conditional [17, 35, 36, 38, 39, 41, 42, 48] settings, even surpassing the quality of GANs [13] in certain situations [9].”
- DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps (2022, background): “Diffusion probabilistic models (DPMs) [1–3] are emerging powerful generative models with promising performance on many tasks, such as image generation [4, 5], video generation [6], text-to-image generation [7], speech synthesis [8, 9] and lossless compression [10].”