Dual Adversarial Inference for Text-to-Image Synthesis

By Qicheng Lao, Mohammad Havaei, Ahmad Pesaranghader, Francis Dutil, Lisa Di-Jorio, Thomas Fevens. 2019

40 citations recorded by the source corpus.

Synthesizing images from a given text description involves engaging two types of information: the content, which includes information explicitly described in the text (e.g., color, composition, etc.), and the style, which is usually not well described in the text (e.g., location, quantity, size, etc.). However, in previous works, it is typically treated as a process of generating images only from the content, i.e., without considering learning meaningful style representations. In this paper, we aim to learn two variables that are disentangled in the latent space, representing content and style respectively. We achieve this by augmenting current text-to-image synthesis frameworks with a dual adversarial inference mechanism. Through extensive experiments, we show that our model learns, in an unsupervised manner, style representations corresponding to certain meaningful information present in the image that are not well described in the text. The new framework also improves the quality of synthesized images when evaluated on Oxford-102, CUB and COCO datasets.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Cross-Modal Contrastive Learning for Text-to-Image Generation (2021, background): “GAN-based models in particular have shown better sample quality [61, 64, 58, 66, 59, 26, 52, 42, 24].”
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis (2020, background): “Many approaches [11], [16], [24], [25], [41], [42] based on Generative Adversarial Networks (GANs) [9] bridge the domain gap by utilizing a discriminator to distinguish the synthesized text-image pair and the ground-truth pair.”
Multimodal Intelligence: Representation Learning, Information Fusion, and Applications (2019, methodology): “learning of representation content and styles as two disentangled variables by using a dual inference mechanism based on cycleconsistency for text-to-image generation [225].”
CPGAN: Content-Parsing Generative Adversarial Networks for Text-to-Image Synthesis (2020, background): “Lao et al. [15] parses the input text by learning two variables that are disentangled in the latent space.”
Adversarial Text-to-Image Synthesis: A Review (2021, methodology): “…a good image (e.g., visual Input Method caption [16], [33], [40], [42], [41], [48], [35], [54], [43], [55], [58], [61], [65], [67], [68], [69], [70], [75], [80], [86], [34], [87], [128] caption + dialogue [93], [95], [99] caption + layout [104], [97], [108], [103] caption + semantic masks…”
CPGAN: Full-Spectrum Content-Parsing Generative Adversarial Networks for Text-to-Image Synthesis (2019, background): “[15] parses the input text by learning two variables that are disentangled in the latent space.”