Object-Driven Text-To-Image Synthesis via Adversarial Training

By Wenbo Li, Pengchuan Zhang, Lei Zhang, Qiuyuan Huang, Xiaodong He, Siwei Lyu, Jianfeng Gao. 2019

324 citations recorded by the source corpus.

In this paper, we propose Object-driven Attentive Generative Adversarial Newtorks (Obj-GANs) that allow attention-driven, multi-stage refinement for synthesizing complex images from text descriptions. With a novel object-driven attentive generative network, the Obj-GAN can synthesize salient objects by paying attention to their most relevant words in the text descriptions and their pre-generated class label. In addition, a novel object-wise discriminator based on the Fast R-CNN model is proposed to provide rich object-wise discrimination signals on whether the synthesized object matches the text description and the pre-generated class label. The proposed Obj-GAN significantly outperforms the previous state of the art in various metrics on the large-scale MS-COCO benchmark, increasing the inception score by 27% and decreasing the FID score by 11%. A thorough comparison between the classic grid attention and the new object-driven attention is provided through analyzing their mechanisms and visualizing their attention layers, showing insights of how the proposed model generates complex scenes in high quality.

Subfields: Computer Vision and Pattern Recognition, Generative Models, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

In Multimodal Learning

Citations in the corpus

Zero-Shot Text-to-Image Generation (2021, methodology): “, 2018), and leveraging additional sources of conditioning information beyond just text (Reed et al., 2016a; Li et al., 2019; Koh et al., 2021).”
Towards Language-Free Training for Text-to-Image Generation (2021, methodology): “We compare L AFITE against a series of competitive systems: AttnGAN [51], Obj-GAN [23], DM-GAN [57], OP-GAN [13], DF-GAN [42] and XMC-GAN [54].”
CogView: Mastering Text-to-Image Generation via Transformers (2021, neutral): “ObjectGAN [29] generated images following a text → boxes → layouts → image process.”
Learning Layout and Style Reconfigurable GANs for Controllable Image Synthesis (2020, methodology): “In [18], [23], [39], a two-step approach is used in image synthesis: generating the semantic layout (class label, bounding boxes and semantic mask) from a text description or a scene graph, and synthesizing images conditioned on the predicted semantic layout and text description (if present).”
Object-Centric Image Generation from Layouts (2020, methodology): “Despite similar interest in scene graph to image retrieval [24,44], and the large improvements in text-to-image synthesis resulting from the DAMSM [60,27], our approach is the ﬁrst to use a scene graph to image retrieval module when training a generative model.”
CLIP-GEN: Language-Free Training of a Text-to-Image Generator with CLIP (2022, background): “Early *Corresponding author. ar X iv :2 20 3. approaches (e.g., DMGAN [45], AttnGAN [39], DF-GAN [35], Obj-GAN [19], OPGAN [19], SD-GAN [40], CPGAN [21], XMC-GAN [41]) that directly generate pixels from the given text embeddings with a convolutional generator have shown promising revolutions to gene”
Vector Quantized Diffusion Model for Text-to-Image Synthesis (2021, methodology): “Other works [22,29] adopt a two-step process which first infer the semantic layout then generate different objects, but this kind of method requires fine-grained object labels, e.”
Improving Text-to-Image Synthesis Using Contrastive Learning (2021, methodology): “Recently, a great number of studies [1, 4, 6, 12, 13, 14, 17, 18, 19, 22, 23, 24, 25, 32, 33, 36, 42, 43, 44, 45, 46, 47, 48] present promising results on the text-to-image synthesis task, most of which make use of GANs as the backbone model.”
StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery (2021, methodology): “Additional supervision was used in other works [37, 25, 20] to further improve the image quality.”
Cross-Modal Contrastive Learning for Text-to-Image Generation (2021, methodology): “Following previous work [58, 27], we use R-precision to assess whether a generated image can be used to retrieve its conditioning description.”
Paint by Word (2021, background): “…generator that explicitly takes a language embedding as input [39]; the image quality of this approach can be improved using multi-stage generators [51, 52, 54], and semantic consistency can be improved through careful design of architectures and training losses [48, 58, 34, 49, 25, 43].”
Text-to-Image Generation Grounded by Fine-Grained User Attention (2020, methodology): “We observed that outputs from leading end-to-end textto-image models [42, 39, 21] leave much to be desired; in particular, their generated images captured a visual gist of the descriptions but lacked well-defined objects and coherent composition.”
Image Synthesis From Reconfigurable Layout and Style (2019, methodology): “Unlike [12, 21] where ground truth masks is adopted to guide learning of shape generator, our model can learn semantic masks in a weakly-supervised manner.”
GLAZE: Protecting Artists from Style Mimicry by Text-to-Image Models (2023, background): “Since Text-to-image generation was first proposed in 2015 [50], a stream of research has proposed newer model architectures and training methods enabling generation of higher-quality images [45, 63, 99, 103, 107].”
Prompt-to-Prompt Image Editing with Cross Attention Control (2022, background): “Numerous works [11, 16, 42, 25, 26, 30, 31, 34, 49, 9, 13, 36] signiﬁcantly advanced the generation of images conditioned on plain text, known as text-to-image synthesis.”
Cycle-Consistent Inverse GAN for Text-to-Image Synthesis (2021, background): “However, IS may fail to reflect the generated image quality in some text-toimage cases [17].”