CLIP-GEN: Language-Free Training of a Text-to-Image Generator with CLIP

By Zihao Wang, Qian He, Xinglong Wu, Zili Yi. 2022

95 citations recorded by the source corpus.

Training a text-to-image generator in the general domain (e.g., Dall.e, CogView) requires huge amounts of paired text-image data, which is too expensive to collect. In this paper, we propose a self-supervised scheme named as CLIP-GEN for general text-to-image generation with the language-image priors extracted with a pre-trained CLIP model. In our approach, we only require a set of unlabeled images in the general domain to train a text-to-image generator. Specifically, given an image without text labels, we first extract the embedding of the image in the united language-vision embedding space with the image encoder of CLIP. Next, we convert the image into a sequence of discrete tokens in the VQGAN codebook space (the VQGAN model can be trained with the unlabeled image dataset in hand). Finally, we train an autoregressive transformer that maps the image tokens from its unified language-vision representation. Once trained, the transformer can generate coherent image tokens based on the text embedding extracted from the text encoder of CLIP upon an input text. Such a strategy enables us to train a strong and general text-to-image generator with large text-free image dataset such as ImageNet. Qualitative and quantitative evaluations verify that our method significantly outperforms optimization-based text-to-image methods in terms of image quality while not compromising the text-image matching. Our method can even achieve comparable performance as flagship supervised models like CogView.

Subfields: Computer Vision and Pattern Recognition, Generative Models, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

In Multimodal Learning

Citations in the corpus

Hierarchical Text-Conditional Image Generation with CLIP Latents (2022, methodology): “Another possibility is to feed the decoder the CLIP text embedding as if it were an image embedding, as previously observed [61, 54].”
VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance (2022, neutral): “In contrast to previous work on this topic [38, 33, 49], we do not perceive creating photo-realistic images or images that could convince a human that they are real photographs as our primary goal.”
Blended Latent Diffusion (2022, methodology): “2022] and text-to-image without training on text data [Ashual et al. 2022; Wang et al. 2022; Zhou et al. 2021].”
KNN-Diffusion: Image Generation via Large-Scale Retrieval (2022, background): “The joint representation was shown to hold a strong semantic alignment between the two modalities, enabling image generation [29, 47], image manipulation [2,34], and image captioning [30].”
Bridging CLIP and StyleGAN through Latent Alignment for Image Editing (2022, unknown): “From then on, many recent works [9, 32, 33, 43] have greatly developed this research direction.”
Orca: A Distributed Serving System for Transformer-Based Generative Models (2022, methodology): “While we use a language model as a driving example throughout the paper and conduct experiments only on language models, generative models in other domains can benefit from our approach as long as the models are based on the Transformer architecture and use the autoregressive generation procedure [1”