VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance

By Katherine Crowson, Stella Biderman, Daniel Kornis, Dashiell Stander, Eric Hallahan, Louis Castricato, Edward Raff. 2022

474 citations recorded by the source corpus.

Generating and editing images from open domain text prompts is a challenging task that heretofore has required expensive and specially trained models. We demonstrate a novel methodology for both tasks which is capable of producing images of high visual quality from text prompts of significant semantic complexity without any training by using a multimodal encoder to guide image generations. We demonstrate on a variety of tasks how using CLIP [37] to guide VQGAN [11] produces higher visual quality outputs than prior, less flexible approaches like DALL-E [38], GLIDE [33] and Open-Edit [24], despite not being trained for the tasks presented. Our code is available in a public repository.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Literature and Literary Theory, Generative Models, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Literature and Literary Theory

In Generative Models

In Multimodal Learning

Citations in the corpus

Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling (2023, background): “Beyond NLP, transformers have also made big splashes as generative models in areas as diverse as text-to-image synthesis (Ramesh et al., 2022; Crowson et al., 2022; Rombach et al., 2022), protein modeling (Jumper et al., 2021; Ahdritz et al., 2022), and computer programming (Chen et al., 2021; Xu…”
InstructPix2Pix: Learning to Follow Image Editing Instructions (2022, methodology): “Recent models have leveraged CLIP [47] embeddings to guide image editing using text [5, 9, 14, 28, 31, 41, 44, 70].”
Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding (2022, methodology): “Figure 3: Comparison between Imagen and DALL-E 2 [56], GLIDE [43], VQ-GAN+CLIP [12] and Latent Diffusion [59] on DrawBench: User preference rates (with 95% confidence intervals) for image-text alignment and image fidelity.”
Emergent and Predictable Memorization in Large Language Models (2023, background): “…architecture that have achieved state-of-the-art results outside of NLP on diverse tasks such as text-to-image synthesis [Ramesh et al., 2022, Crowson et al., 2022, Rombach et al., 2022], code generation [Chen et al., 2021, Xu et al., 2022, Fried et al., 2022], and protein modeling [Jumper…”
Bridging CLIP and StyleGAN through Latent Alignment for Image Editing (2022, unknown): “From then on, many recent works [9, 32, 33, 43] have greatly developed this research direction.”
Prompt Engineering for Text-Based Generative Art (2022, background): “Inputs for text-to-image synthesis systems, such as VQGAN+CLIP [9], CLIP guided diffusion [7], PixelDraw [34], and ruDALL-E [31], are provided as textual input prompts.”
KNN-Diffusion: Image Generation via Large-Scale Retrieval (2022, background): “The majority of previous works in the task of image manipulation either rely on user-provided masks (Nichol et al., 2021; Avrahami et al., 2022b;a), or are limited to global editing (Crowson et al., 2022; Kim et al., 2022).”
Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation (2022, methodology): “Note that VQGAN-CLIP and DiffuseIT directly use the evaluation metrics in their objective (CLIP loss in [10] and DINO self-similarity in [25]), which explains their respective scores in these metrics.”
Inversion-based Style Transfer with Diffusion Models (2022, methodology): “VQGAN-CLIP [5] uses the CLIP-guided VQGAN [12] to generate artistic images of various styles from text prompts.”
MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editing (2023, methodology): “(2) VQGAN-CLIP [8] fine-tunes VQGAN [12] with CLIP embedding [30] similarity between generated image and target text.”
StyleGAN-T: Unlocking the Power of GANs for Fast Large-Scale Text-to-Image Synthesis (2023, methodology): “CLIP has been used in prior work to guide a pretrained generator during synthesis (Nichol et al., 2022; Crowson et al., 2022; Liu et al., 2021).”
DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation (2022, background): “Crowson et al. [14] use VQ-GAN [18] and train over more diverse data to alleviate this concern.”
InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation (2023, background): “52, 53, 54], GAN inversion [55, 56], GAN-based approaches [57], and diffusion models [8, 9, 58, 59].”
Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models (2023, methodology): “In the context of image editing, while most methods are generally limited to global edits [Chefer et al. 2022a; Crowson et al. 2022; Gal et al. 2022b; Kwon and Ye 2022], several works introduce a user-provided mask to specify the region that should be altered [Avrahami et al. 2022a; Bau et al.…”
The Creativity of Text-to-Image Generation (2022, methodology): “The image was created with VQGAN-CLIP [8] with the prompt”
Generating Human Motion from Textual Descriptions with Discrete Representations (2023, methodology): “We leverage CLIP [55] to extract text embedding c, which has shown its effectiveness in relevant tasks [14, 57, 65].”
Prompt-to-Prompt Image Editing with Cross Attention Control (2022, background): “, no masks) are limited to global editing (Crowson et al., 2022; Kwon & Ye, 2021), Bar-Tal et al.”