Towards Language-Free Training for Text-to-Image Generation

By Yufan Zhou, Ruiyi Zhang, Changyou Chen, Chunyuan Li, Chris Tensmeyer, Tong Yu, Jiuxiang Gu, Jinhui Xu, Tong Sun. 2021

229 citations recorded by the source corpus.

One of the major challenges in training text-to-image generation models is the need of a large number of highquality image-text pairs. While image samples are often easily accessible, the associated text descriptions typically require careful human captioning, which is particularly time- and cost-consuming. In this paper, we propose the first work to train text-to-image generation models without any text data. Our method leverages the well-aligned multi-modal semantic space of the powerful pre-trained CLIP model: the requirement of text-conditioning is seamlessly alleviated via generating text features from image features. Extensive experiments are conducted to illustrate the effectiveness of the proposed method. We obtain state-of-the-art results in the standard text-to-image generation tasks. Importantly, the proposed language-free model outperforms most existing models trained with full image-text pairs. Furthermore, our method can be applied in fine-tuning pretrained models, which saves both training time and cost in training text-to-image generation models. Our pre-trained model obtains competitive results in zero-shot text-to-image generation on the MS-COCO dataset, yet with around only 1% of the model size and training data size relative to the recently proposed large DALL-E model.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

High-Resolution Image Synthesis with Latent Diffusion Models (2021, methodology): “Evaluation of text-conditional image synthesis on the MS-COCO [50] dataset: Our model outperforms autoregressive [17, 64] and GAN-based [105] methods by a significant margin when using 250 DDIM [81] steps .”
Hierarchical Text-Conditional Image Generation with CLIP Latents (2022, background): “[61] condition GAN models on randomly perturbed CLIP image embeddings, finding that these models can generalize to CLIP text embeddings to produce text-conditional images.”
GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models (2021, neutral): “(Zhu et al., 2019) 32.64 DF-GAN (Tao et al., 2020) 21.42 DM-GAN + CL (Ye et al., 2021) 20.79 XMC-GAN (Zhang et al., 2021) 9.33 LAFITE (Zhou et al., 2021) 8.12 DALL-E (Ramesh et al., 2021) ∼ 28 LAFITE (Zhou et al., 2021) 26.94 GLIDE 12.24 GLIDE (Validation ﬁltered)”
GLIGEN: Open-Set Grounded Text-to-Image Generation (2023, methodology): “For this, we train and evaluate on the COCO2014 [41] dataset, which is a standard benchmark used in the text2img literature [51, 56, 65,70,82], and evaluate how the different types of grounding instructions impact our model’s performance.”
Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding (2022, methodology): “The COCO [38] validation set is the standard benchmark for evaluating text-to-image models for both the supervised [85, 22] and the zero-shot setting [55, 43].”
Diffusion Probabilistic Modeling for Video Generation (2022, background): “Videos can be generated based on side information of various types, such as images [30,31], text [32–37] or other videos [38].”
StyleGAN-T: Unlocking the Power of GANs for Fast Large-Scale Text-to-Image Synthesis (2023, methodology): “At high resolution, StyleGAN-T still signiﬁcantly outperforms LAFITE but lags behind DMs and ARMs in terms of FID.”
AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning (2023, background): “In recent years, text-to-image (T2I) diffusion models have gained much popularity both in and beyond the research community, ben-eﬁted by the large-scale text-image paired data [26] and the power of diffusion models [5, 11].”
Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis (2022, methodology): “More recently, LAFITE (Zhou et al., 2022) enables language-free training by constructing pseudo-image-text feature pairs using CLIP (Radford et al., 2021).”