Semantics-Enhanced Adversarial Nets for Text-to-Image Synthesis

By Hongchen Tan, Xiuping Liu, Xin Li, Yi Zhang, Baocai Yin. 2019

80 citations recorded by the source corpus.

This paper presents a new model, Semantics-enhanced Generative Adversarial Network (SEGAN), for fine-grained text-to-image generation. We introduce two modules, a Semantic Consistency Module (SCM) and an Attention Competition Module (ACM), to our SEGAN. The SCM incorporates image-level semantic consistency into the training of the Generative Adversarial Network (GAN), and can diversify the generated images and improve their structural coherence. A Siamese network and two types of semantic similarities are designed to map the synthesized image and the groundtruth image to nearby points in the latent semantic feature space. The ACM constructs adaptive attention weights to differentiate keywords from unimportant words, and improves the stability and accuracy of SEGAN. Extensive experiments demonstrate that our SEGAN significantly outperforms existing state-of-the-art methods in generating photo-realistic images. All source codes and models will be released for comparative study.

Subfields: Computer Vision and Pattern Recognition, Generative Models, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Trevor Darrell
2. Dacheng Tao
3. Luc Van Gool
4. Ming-Hsuan Yang
5. Xiaogang Wang
6. Andrew Zisserman
7. Jitendra Malik
8. Andrea Vedaldi

In Generative Models

1. Trevor Darrell
2. Tobias Hinz
3. Jing Yu Koh
4. Feng Lu
5. Wenjie Pei
6. Jiadong Liang
7. Han Zhang
8. Samaneh Azadi

In Multimodal Learning

1. Trevor Darrell
2. Anna Rohrbach
3. Jing Yu Koh
4. Han Zhang
5. Jason Baldridge
6. Yinfei Yang
7. Feng Lu
8. Wenjie Pei

Citations in the corpus

• Cross-Modal Contrastive Learning for Text-to-Image Generation (2021, background): “GAN-based models in particular have shown better sample quality [61, 64, 58, 66, 59, 26, 52, 42, 24].”
• CPGAN: Content-Parsing Generative Adversarial Networks for Text-to-Image Synthesis (2020, background): “Most existing methods [4, 9–11, 17, 32, 34, 39, 43, 44, 49, 50, Fig.”
• Benchmark for Compositional Text-to-Image Synthesis (2021, background): “Other approaches explored introducing memory mechanisms [63, 24], cycle-consistency constraint [37, 17], siamese networks [57, 47], and prior knowledge [36, 4] to improve the text-to-image synthesis performance.”
• Adversarial Text-to-Image Synthesis: A Review (2021, background): “The authors of SEGAN [55] proposed an attention competition module to focus only on key-words instead of defining an attention weight for each word in the sentence (as is done in AttnGAN).”
• CPGAN: Full-Spectrum Content-Parsing Generative Adversarial Networks for Text-to-Image Synthesis (2019, methodology): “…Reed [23] 7.88 ± 0.07 − StackGAN [34] 8.45 ± 0.03 − Infer [8] 11.46 ± 0.09 − SD-GAN [32] 35.69 ± 0.50 − MirrorGAN [21] 26.47 ± 0.41 − SEGAN [28] 27.86 ± 0.31 − DMGAN [37] 30.49 ± 0.57 88.56% AttnGAN [31] 25.89 ± 0.47 82.98% objGAN[12] 30. ages by these three models and vote for the best…”
• KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis (2020, methodology): “SEGAN [13] introduces a new contrastive loss and a Semantic Consistency Module (SCM) to better align the synthesized image and the ground truth in feature space.”