Adversarial Learning of Semantic Relevance in Text to Image Synthesis

By Miriam Cha, Youngjune Gwon, H. T. Kung. 2018

60 citations recorded by the source corpus.

We describe a new approach that improves the training of generative adversarial nets (GANs) for synthesizing diverse images from a text input. Our approach is based on the conditional version of GANs and expands on previous work leveraging an auxiliary task in the discriminator. Our generated images are not limited to certain classes and do not suffer from mode collapse while semantically matching the text input. A key to our training methods is how to form positive and negative training examples with respect to the class label of a given image. Instead of selecting random training examples, we perform negative sampling based on the semantic distance from a positive example in the class. We evaluate our approach using the Oxford-102 flower dataset, adopting the inception score and multi-scale structural similarity index (MS-SSIM) metrics to assess discriminability and diversity of the generated images. The empirical results indicate greater diversity in the generated images, especially when we gradually select more negative training examples closer to a positive example in the semantic space.

Subfields: Computer Vision and Pattern Recognition, Law, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Xiaodong He
2. Dacheng Tao
3. Mohit Bansal
4. Trevor Darrell
5. Luc Van Gool
6. Li Deng
7. Jianfeng Gao
8. Xiaogang Wang

In Law

1. Stefan Wermter
2. Stefan Heinrich
3. Tobias Hinz

In Generative Models

1. Tobias Hinz
2. Stefan Wermter
3. Stefan Heinrich
4. Feng Lu
5. Wenjie Pei
6. Jiadong Liang
7. Xiulong Yang
8. Xiaodong He

Citations in the corpus

• Improving Text-to-Image Synthesis Using Contrastive Learning (2021, methodology): “Recently, a great number of studies [1, 4, 6, 12, 13, 14, 17, 18, 19, 22, 23, 24, 25, 32, 33, 36, 42, 43, 44, 45, 46, 47, 48] present promising results on the text-to-image synthesis task, most of which make use of GANs as the backbone model.”
• CPGAN: Content-Parsing Generative Adversarial Networks for Text-to-Image Synthesis (2020, background): “Text-SeGAN [5] focuses on devising a specific discriminator to regress the semantic relevance between text and image.”
• Adversarial Text-to-Image Synthesis: A Review (2021, methodology): “Instead of randomly sampling a mismatching negative image sample, in Text-SeGAN [65] several strategies based on curriculum learning [66] are introduced to select negative samples with gradually increasing semantic difficulty.”
• Semantic Object Accuracy for Generative Text-to-Image Synthesis (2019, methodology): “[17] modify the sampling procedure during training to obtain a curriculum of mis-matching caption-image pairs and introduce an auxiliary classifier that specifically predicts the semantic consistency of a given caption-image pair.”
• Dual Adversarial Inference for Text-to-Image Synthesis (2019, background): “Several attempts have been made to leverage additional available information, such as object location [27], class label [5, 2], attention extracted from word features [30, 24] and text regeneration [24].”
• Multimodal Intelligence: Representation Learning, Information Fusion, and Applications (2019, methodology): “GAN (Text-SeGAN) trains classifiers by using regression tasks to estimate the semantic relevance between images and text [211].”
• CPGAN: Full-Spectrum Content-Parsing Generative Adversarial Networks for Text-to-Image Synthesis (2019, background): “xt-visual co-embeddings to replace input text with corresponding visual features. Lao et al. [15] parses the input text by learning two variables that are disentangled in the latent space. Text-SeGAN [5] focuses on devising a specic discriminator to regress the semantic relevance between text and im”