Cross-Modal Contrastive Learning for Text-to-Image Generation

By Han Zhang, Jing Yu Koh, Jason Baldridge, Honglak Lee, Yinfei Yang. 2021

451 citations recorded by the source corpus.

The output of text-to-image synthesis systems should be coherent, clear, photo-realistic scenes with high semantic fidelity to their conditioned text descriptions. Our Cross-Modal Contrastive Generative Adversarial Network (XMC-GAN) addresses this challenge by maximizing the mutual information between image and text. It does this via multiple contrastive losses which capture inter-modality and intra-modality correspondences. XMC-GAN uses an attentional self-modulation generator, which enforces strong text-image correspondence, and a contrastive discriminator, which acts as a critic as well as a feature encoder for contrastive learning. The quality of XMC-GAN’s output is a major step up from previous models, as we show on three challenging datasets. On MS-COCO, not only does XMC-GAN improve state-of-the-art FID from 24.70 to 9.33, but– more importantly–people prefer XMC-GAN by 77.3% for image quality and 74.1% for image-text alignment, compared to three other recent models. XMC-GAN also generalizes to the challenging Localized Narratives dataset (which has longer, more detailed descriptions), improving state-of-the-art FID from 48.70 to 14.12. Lastly, we train and evaluate XMC-GAN on the challenging Open Images data, establishing a strong benchmark FID score of 26.91.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Representation & Self-Supervised Learning

Citations in the corpus

Hierarchical Text-Conditional Image Generation with CLIP Latents (2022, background): “Many previous works have trained GANs [21] on publicly available image captioning datasets to produce text-conditional image samples [56, 63, 49, 58, 57].”
Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors (2022, methodology): “Text-to-image generation [64, 72, 54, 65, 67, 45, 12, 41, 70] focuses on generating images from standalone text descriptions.”
GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models (2021, methodology): “(Zhu et al., 2019) 32.64 DF-GAN (Tao et al., 2020) 21.42 DM-GAN + CL (Ye et al., 2021) 20.79 XMC-GAN (Zhang et al., 2021) 9.33 LAFITE (Zhou et al., 2021) 8.12 DALL-E (Ramesh et al., 2021) ∼ 28 LAFITE (Zhou et al., 2021) 26.94 GLIDE 12.24 GLIDE (Validation ﬁltered)”
CLIP-GEN: Language-Free Training of a Text-to-Image Generator with CLIP (2022, background): “Early *Corresponding author. ar X iv :2 20 3. approaches (e.g., DMGAN [45], AttnGAN [39], DF-GAN [35], Obj-GAN [19], OPGAN [19], SD-GAN [40], CPGAN [21], XMC-GAN [41]) that directly generate pixels from the given text embeddings with a convolutional generator have shown promising revolutions to gene”
Improving Text-to-Image Synthesis Using Contrastive Learning (2021, methodology): “We compute the contrastive losses of the caption-caption pair and the fake-fake pair, which are complementary to the contrastive losses in XMC-GAN.”
Scaling Autoregressive Models for Content-Rich Text-to-Image Generation (2022, methodology): “On MS-COCO and PartiPrompts, we also obtain human side-by-side evaluations for Parti 20B to compare with a strong retrieval baseline, as well as the XMC-GAN model [47], which has the best FID out of all publicly available models at the time of writing.”
Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding (2022, methodology): “[84] Han Zhang, Jing Yu Koh, Jason Baldridge, Honglak Lee, and Yinfei Yang.”
T2I-CompBench: A Comprehensive Benchmark for Open-world Compositional Text-to-image Generation (2023, unknown): “Early works [18, 19, 20, 21, 22, 23] explore different network architectures and loss functions based on generative adversarial networks (GAN) [24].”
An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion (2022, background): “, 2018) or cross-modal contrastive approaches (Zhang et al., 2021; Ye et al., 2021).”
Make-A-Video: Text-to-Video Generation without Text-Video Data (2022, background): “Later GAN variants have focused on progressive generation (Zhang et al., 2017; Hong et al., 2018), or better text-image alignment (Xu et al., 2018; Zhang et al., 2021).”
Benchmark for Compositional Text-to-Image Synthesis (2021, background): “Recent works explored a transformer-based vision-language representation model [5], cross-modal constrastive learning [59], and transformer and discrete VAE [39] for text-to-image synthesis.”
Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generation (2023, unknown): “, 2018) and (Zhang et al., 2021) seek to improve text-image alignment.”
Training-Free Layout Control with Cross-Attention Guidance (2023, background): “For several years, generative adversarial networks (GANs) [10] have been the dominant approach in image generation from textual prompts [30, 46, 47, 38, 40, 45].”