Photographic Text-to-Image Synthesis with a Hierarchically-Nested Adversarial Network

By Zizhao Zhang, Yuanpu Xie, Lin F. Yang. 2018

323 citations recorded by the source corpus.

This paper presents a novel method to deal with the challenging task of generating photographic images conditioned on semantic image descriptions. Our method introduces accompanying hierarchical-nested adversarial objectives inside the network hierarchies, which regularize mid-level representations and assist generator training to capture the complex image statistics. We present an extensile single-stream generator architecture to better adapt the jointed discriminators and push generated images up to high resolutions. We adopt a multi-purpose adversarial loss to encourage more effective image and text information usage in order to improve the semantic consistency and image fidelity simultaneously. Furthermore, we introduce a new visual-semantic similarity measure to evaluate the semantic consistency of generated images. With extensive experimental validation on three public datasets, our method significantly improves previous state of the arts on all datasets over different evaluation metrics.

Subfields: Computer Vision and Pattern Recognition, Generative Models, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

In Multimodal Learning

Citations in the corpus

Improving Text-to-Image Synthesis Using Contrastive Learning (2021, methodology): “Recently, a great number of studies [1, 4, 6, 12, 13, 14, 17, 18, 19, 22, 23, 24, 25, 32, 33, 36, 42, 43, 44, 45, 46, 47, 48] present promising results on the text-to-image synthesis task, most of which make use of GANs as the backbone model.”
AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks (2017, background): “Compared with other deep generative models, Generative Adversarial Networks (GANs) [6] have shown great performance for generating sharper samples [17, 3, 23, 13, 10, 35, 24, 34, 39, 40].”
Diffsound: Discrete Diffusion Model for Text-to-Sound Generation (2022, background): “In the past few years, Generative Adversarial Networks (GANs) have shown promising results in image generation [16], [17], [18], speech synthesis [6], [19], [20] and music genera-”
Adversarial Latent Autoencoders (2020, background): “HDGAN [57] improves by incorporating hierarchically-nested adversarial objectives inside the network hierarchy.”
Paint by Word (2021, background): “The second approach is to train a conditional generator that explicitly takes a language embedding as input [35]; the image quality of this approach can be improved using multi-stage generators [47, 48, 50], and semantic consistency can be improved through careful design of architectures and trainin”
Semantics Disentangling for Text-To-Image Generation (2019, methodology): “Following prior arts on text-to-image generation [37, 40, 41, 14, 42, 18], we apply the numerical assessment approach “inception score” [31] for quantitative evaluation.”
MirrorGAN: Learning Text-To-Image Generation by Redescription (2019, background): “proposed training a T2I generator with hierarchically nested adversarial objectives [42].”
Consistency Regularization for Generative Adversarial Networks (2019, background): “Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) have recently demonstrated impressive results on image-synthesis benchmarks (Radford et al., 2016; Zhang et al., 2017; Miyato & Koyama, 2018; Zhang et al., 2018; Brock et al., 2018; Karras et al., 2019).”
Generating Multiple Objects at Spatially Distinct Locations (2019, methodology): “In our work we mostly use the StackGAN (Zhang et al., 2018a) and AttnGAN (Xu et al., 2018b) frameworks which use a modified objective function taking into consideration the additional conditional information and provided image captions.”
Blended Diffusion for Text-driven Editing of Natural Images (2021, background): “Recent works that leverage the tremendous progress in vision-language models and datadriven image generation have demonstrated that text-based interfaces for image creation and manipulation are now finally within reach [12, 23, 29, 30, 38, 39, 41, 47, 51, 57].”
Translating and Segmenting Multimodal Medical Volumes with Cycle- and Shape-Consistency Generative Adversarial Network (2018, background): “GANs have attracted wide interests in helping addressing such tasks to generate high-quality, less blurry results [10, 1, 2, 41].”
Adversarial Learning of Semantic Relevance in Text to Image Synthesis (2018, methodology): “As used in other approaches (Reed et al. 2016b; Dash et al. 2017; Zhang et al. 2016; Cha, Gwon, and Kung 2017; Zhang, Xie, and Yang 2018), choose any image outside the class of the reference image as a random negative.”
VocGAN: A High-Fidelity Real-time Vocoder with a Hierarchically-nested Adversarial Network (2020, methodology): “Inspired by the work of [15], we extend the generator to output multiple waveforms at different scales.”
IntroVAE: Introspective Variational Autoencoders for Photographic Image Synthesis (2018, methodology): “StackGAN-v2 [43] and HDGAN [44] adopt multi-scale discriminators in a treelike structure.”
CKD: Cross-Task Knowledge Distillation for Text-to-Image Synthesis (2020, methodology): “network has become a highlighted research topic, such as image captioning [6]–[8] and text-to-image synthesis [9]–[11].”