AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks

By Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, Xiaodong He. 2017

1,976 citations recorded by the source corpus.

In this paper, we propose an Attentional Generative Adversarial Network (AttnGAN) that allows attention-driven, multi-stage refinement for fine-grained text-to-image generation. With a novel attentional generative network, the AttnGAN can synthesize fine-grained details at different sub-regions of the image by paying attentions to the relevant words in the natural language description. In addition, a deep attentional multimodal similarity model is proposed to compute a fine-grained image-text matching loss for training the generator. The proposed AttnGAN significantly outperforms the previous state of the art, boosting the best reported inception score by 14.14% on the CUB dataset and 170.25% on the more challenging COCO dataset. A detailed analysis is also performed by visualizing the attention layers of the AttnGAN. It for the first time shows that the layered attentional GAN is able to automatically select the condition at the word level for generating different parts of the image.

Subfields: Computer Vision and Pattern Recognition, Generative Models, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

In Multimodal Learning

Citations in the corpus

Hierarchical Text-Conditional Image Generation with CLIP Latents (2022, background): “Many previous works have trained GANs [21] on publicly available image captioning datasets to produce text-conditional image samples [56, 63, 49, 58, 57].”
Zero-Shot Text-to-Image Generation (2021, methodology): “This process can be seen as a kind of language-guided search (Andreas et al., 2017), and is also similar to the auxiliary text-image matching loss proposed by Xu et al. (2018).”
Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors (2022, methodology): “Model FID ↓ FID ↓ Image Photo-Text (ﬁlt.) quality realism alignment AttnGAN [64] 35 .”
GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models (2021, background): “Model FID Zero-shot FID AttnGAN (Xu et al., 2017) 35.”
Towards Language-Free Training for Text-to-Image Generation (2021, methodology): “We compare L AFITE against a series of competitive systems: AttnGAN [51], Obj-GAN [23], DM-GAN [57], OP-GAN [13], DF-GAN [42] and XMC-GAN [54].”
CogView: Mastering Text-to-Image Generation via Transformers (2021, methodology): “AttnGAN [51] used attention on words to focus on the corresponding subregion.”
High-Resolution Complex Scene Synthesis with Transformers (2021, background): “in class-conditional [4, 24], text-conditional [23, 34] or pixelwise conditional settings [20, 33, 30].”
Learning Layout and Style Reconfigurable GANs for Controllable Image Synthesis (2020, background): “Spatial layouts are usually used as intermediate representations for other conditional image synthesis tasks such as text-to-image [17], [21] and scene-graph-toimage [18], [19].”
Object-Centric Image Generation from Layouts (2020, methodology): “Despite similar interest in scene graph to image retrieval [24,44], and the large improvements in text-to-image synthesis resulting from the DAMSM [60,27], our approach is the ﬁrst to use a scene graph to image retrieval module when training a generative model.”
Semantic Image Synthesis With Spatially-Adaptive Normalization (2019, background): “Researchers have explored various models for generating images based on text [18,41,49, 52].”
Improving Text-to-Image Synthesis Using Contrastive Learning (2021, methodology): “We evaluate our approach over two popular base models, AttnGAN [42] and DM-GAN [48] on datasets CUB [37] and COCO [20].”
StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery (2021, background): “AttnGAN [52] incorporated an attention mechanism between the text and image features.”
DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-To-Image Synthesis (2019, methodology): “Conditioning on the global sentence vector may result in low-quality images, AttnGAN [28] refines the images to high-resolution ones by leveraging the attention mechanism.”
Towards Coherent and Cohesive Long-form Text Generation (2018, methodology): “Our design of these two discriminators is inspired by the Deep Structured Semantic Model (DSSM) which was originally developed to measure the semantic similarity between two texts (Huang et al., 2013; Gao et al., 2014; Palangi et al., 2016; Xu et al., 2018).”
Scaling Autoregressive Models for Content-Rich Text-to-Image Generation (2022, methodology): “Later works improved upon generation quality by introducing progressive refinement [68, 69] and using cross-modal attention mechanisms [47, 54].”