Semantics Disentangling for Text-To-Image Generation

By Guojun Yin, Lu Sheng, Nenghai Yu, Xiaogang Wang, Jing Shao. 2019

208 citations recorded by the source corpus.

Synthesizing photo-realistic images from text descriptions is a challenging problem. Previous studies have shown remarkable progresses on visual quality of the generated images. In this paper, we consider semantics from the input text descriptions in helping render photo-realistic images. However, diverse linguistic expressions pose challenges in extracting consistent semantics even they depict the same thing. To this end, we propose a novel photo-realistic text-to-image generation model that implicitly disentangles semantics to both fulfill the high-level semantic consistency and low-level semantic diversity. To be specific, we design (1) a Siamese mechanism in the discriminator to learn consistent high-level semantics, and (2) a visual-semantic embedding strategy by semantic-conditioned batch normalization to find diverse low-level semantics. Extensive experiments and ablation studies on CUB and MS-COCO datasets demonstrate the superiority of the proposed method in comparison to state-of-the-art methods.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Object-Centric Image Generation from Layouts (2020, methodology): “Text-based image generation approaches are also not immune to small perturbations in text leading to quite different images (Yin et al. 2019).”
CLIP-GEN: Language-Free Training of a Text-to-Image Generator with CLIP (2022, background): “Early *Corresponding author. ar X iv :2 20 3. approaches (e.g., DMGAN [45], AttnGAN [39], DF-GAN [35], Obj-GAN [19], OPGAN [19], SD-GAN [40], CPGAN [21], XMC-GAN [41]) that directly generate pixels from the given text embeddings with a convolutional generator have shown promising revolutions to gene”
Improving Text-to-Image Synthesis Using Contrastive Learning (2021, methodology): “Compared with the Siamese structure in SD-GAN, our approach is derived from the recent development of the contrastive learning paradigm, therefore, it has the advantage of better performance and less computational cost.”
Cross-Modal Contrastive Learning for Text-to-Image Generation (2021, methodology): “Figure 3 shows human evaluations comparing XMC-GAN to three recent strong models: CPGAN [28], SD-GAN [59], and OP-GAN [17].”
Connecting Vision and Language with Localized Narratives (2019, background): “In Out - - Image captioning [54, 58, 55] Out In - - Text-to-image Generation [45, 50, 59] In Out - Out Dense image captioning [23, 60, 27], Dense Relational Captioning [27] In Out - In Controllable and Grounded Captioning [11] In In - Out Phrase grounding [14] In In + Out - - Visual Question Answeri”
Paint by Word (2021, background): “…generator that explicitly takes a language embedding as input [39]; the image quality of this approach can be improved using multi-stage generators [51, 52, 54], and semantic consistency can be improved through careful design of architectures and training losses [48, 58, 34, 49, 25, 43].”
Benchmark for Compositional Text-to-Image Synthesis (2021, background): “Other approaches explored introducing memory mechanisms [63, 24], cycle-consistency constraint [37, 17], siamese networks [57, 47], and prior knowledge [36, 4] to improve the text-to-image synthesis performance.”
Semantic Object Accuracy for Generative Text-to-Image Synthesis (2019, background): “[21] extend on this and also consider semantics from the text description during the generation process.”
CPGAN: Content-Parsing Generative Adversarial Networks for Text-to-Image Synthesis (2020, methodology): “Most existing methods [4, 9–11, 17, 32, 34, 39, 43, 44, 49, 50, Fig.”
Adversarial Text-to-Image Synthesis: A Review (2021, background): “Each branch operates on a diﬀerent input, Simpliﬁed SD-GAN [61] architecture.”
MixNMatch: Multifactor Disentanglement and Encoding for Conditional Image Generation (2019, background): “Conditional image generation has various forms, including models conditioned on a class label [29, 28, 3] or text input [33, 48, 42, 47].”
DF-GAN: A Simple and Effective Baseline for Text-to-Image Synthesis (2020, methodology): “Although impressive results have been presented by previous works [9,19,21,32,33,51,60], there still remain three problems.”
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis (2020, background): “Besides, the SD-GAN contains many Siamese branches, which is much more complex than KT-GAN.”
Efficient Neural Architecture for Text-to-Image Synthesis (2020, background): “Finally, SD-GAN [18] proposes a siamese multi-stage networks structure that is intended to make generated images consistent across a variety of descriptions.”
DF-GAN: Deep Fusion Generative Adversarial Networks for Text-to-Image Synthesis (2020, methodology): “The Stacked GANs family [7], [10], [8], [9], [12], [11], [13], [14], [15], [16], [17], [18] makes a summation of all discriminator losses as the whole discriminator loss for training end to end.”