MirrorGAN: Learning Text-To-Image Generation by Redescription

By Tingting Qiao, Jing Zhang, Duanqing Xu, Dacheng Tao. 2019

614 citations recorded by the source corpus.

Generating an image from a given text description has two goals: visual realism and semantic consistency. Although significant progress has been made in generating high-quality and visually realistic images using generative adversarial networks, guaranteeing semantic consistency between the text description and visual content remains very challenging. In this paper, we address this problem by proposing a novel global-local attentive and semantic-preserving text-to-image-to-text framework called MirrorGAN. MirrorGAN exploits the idea of learning text-to-image generation by redescription and consists of three modules: a semantic text embedding module (STEM), a global-local collaborative attentive module for cascaded image generation (GLAM), and a semantic text regeneration and alignment module (STREAM). STEM generates word- and sentence-level embeddings. GLAM has a cascaded architecture for generating target images from coarse to fine scales, leveraging both local word attention and global sentence attention to progressively enhance the diversity and semantic consistency of the generated images. STREAM seeks to regenerate the text description from the generated image, which semantically aligns with the given text description. Thorough experiments on two public benchmark datasets demonstrate the superiority of MirrorGAN over other representative state-of-the-art methods.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Improving Text-to-Image Synthesis Using Contrastive Learning (2021, methodology): “Recently, a great number of studies [1, 4, 6, 12, 13, 14, 17, 18, 19, 22, 23, 24, 25, 32, 33, 36, 42, 43, 44, 45, 46, 47, 48] present promising results on the text-to-image synthesis task, most of which make use of GANs as the backbone model.”
CommonGen: A Constrained Text Generation Challenge for Generative Commonsense Reasoning (2020, background): “nguage generation models. We argue that such an ability of recovering natural scenes of daily life can beneﬁt a wide range of natural language generation (NLG) tasks including image/video captioning (Qiao et al., 2019;Wang et al.,2019), scene-based visual reasoning and VQA (Hudson and Manning,2019),”
Paint by Word (2021, background): “The second approach is to train a conditional generator that explicitly takes a language embedding as input [35]; the image quality of this approach can be improved using multi-stage generators [47, 48, 50], and semantic consistency can be improved through careful design of architectures and trainin”
Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators (2023, methodology): “These include StackGAN [46], AttnGAN [43], and MirrorGAN [24], which further improve image quality and diversity by introducing novel architectures and attention mechanisms.”
Learn, Imagine and Create: Text-to-Image Generation from Prior Knowledge (2019, methodology): “As shown in the current state-of-the-art [37, 46, 20], a carefully designed adversarial loss function is essential for stable training and optimal performance.”
CPGAN: Content-Parsing Generative Adversarial Networks for Text-to-Image Synthesis (2020, methodology): “Most existing methods [4, 9–11, 17, 32, 34, 39, 43, 44, 49, 50, Fig.”
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis (2020, methodology): “However, such a discriminator alone is usually insufficient to model underlying semantic consistency between text and image [23], and consequently, results in semantic or structural errors in synthesized images (see Figure 1, the “Direct T2I” column).”
RiFeGAN: Rich Feature Generation for Text-to-Image Synthesis From Prior Knowledge (2020, background): “[20] introduce a MirrorGAN to connect the dual tasks, text-to-image synthesis and text captioning, and constrain the regenerated caption aligning with the given caption.”
Improving Augmentation and Evaluation Schemes for Semantic Image Synthesis (2020, background): “Conditional GANs [16, 34] can generate images via side information, such as class labels [72, 5], textual descriptions [45, 43, 65, 73, 74], images [23, 77, 22], scene graphs [24, 59], or semantic label maps [41, 56, 78, 80, 38] as in this work.”
Prompt-to-Prompt Image Editing with Cross Attention Control (2022, background): “Numerous works [11, 16, 42, 25, 26, 30, 31, 34, 49, 9, 13, 36] signiﬁcantly advanced the generation of images conditioned on plain text, known as text-to-image synthesis.”
Adversarial Text-to-Image Synthesis: A Review (2021, methodology): “Figure 9: Simpliﬁed MirrorGAN [68] architecture.”
A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications (2020, background): “Field Subfield Method Super-resolution SRGAN [63], ESRGAN [64], Cycle-in-Cycle GANs [65], SRDGAN [66], TGAN [67] DR-GAN [68], TP-GAN [69], PG2 [70], PSGAN [71], Image synthesis and manipulation APDrawingGAN [72], IGAN [73], Image processing and computer vision introspective adversarial networks [74]”
GODIVA: Generating Open-DomaIn Videos from nAtural Descriptions (2021, background): “2 Text-to-image generation Text-to-image generation has been widely researched in recent years [17].”
DF-GAN: Deep Fusion Generative Adversarial Networks for Text-to-Image Synthesis (2020, methodology): “The Stacked GANs family [7], [10], [8], [9], [12], [11], [13], [14], [15], [16], [17], [18] makes a summation of all discriminator losses as the whole discriminator loss for training end to end.”
More Control for Free! Image Synthesis with Semantic Diffusion Guidance (2021, methodology): “Various methods have been studied, proposing many different text-adaptive architectures and loss functions to enforce better semantic alignment between the input text and generated image, resulting in significant improvements in editing quality and correctness [55,60,51,27,11,61,33,35,17,28,7,23,40]”