StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networks

By Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, Dimitris N. Metaxas. 2017

1,190 citations recorded by the source corpus.

Although Generative Adversarial Networks (GANs) have shown remarkable success in various tasks, they still face challenges in generating high quality images. In this paper, we propose Stacked Generative Adversarial Networks (StackGANs) aimed at generating high-resolution photo-realistic images. First, we propose a two-stage generative adversarial network architecture, StackGAN-v1, for text-to-image synthesis. The Stage-I GAN sketches the primitive shape and colors of a scene based on a given text description, yielding low-resolution images. The Stage-II GAN takes Stage-I results and the text description as inputs, and generates high-resolution images with photo-realistic details. Second, an advanced multi-stage generative adversarial network architecture, StackGAN-v2, is proposed for both conditional and unconditional generative tasks. Our StackGAN-v2 consists of multiple generators and multiple discriminators arranged in a tree-like structure; images at multiple scales corresponding to the same scene are generated from different branches of the tree. StackGAN-v2 shows more stable training behavior than StackGAN-v1 by jointly approximating multiple distributions. Extensive experiments demonstrate that the proposed stacked generative adversarial networks significantly outperform other state-of-the-art methods in generating photo-realistic images.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Zero-Shot Text-to-Image Generation (2021, neutral): “These include improving the generative model architecture with modifications like multi-scale generators (Zhang et al., 2017; 2018), integrating attention and auxiliary losses (Xu et al., 2018), and leveraging additional sources of conditioning information beyond just text (Reed et al., 2016a; Li…”
Semantic Image Synthesis With Spatially-Adaptive Normalization (2019, methodology): “Recent methods directly learn the mapping using neural networks [4, 7, 20, 39, 40, 45, 46, 47].”
Vector Quantized Diffusion Model for Text-to-Image Synthesis (2021, methodology): “We qualitatively compare the proposed method with several state-of-the-art methods, including some GAN-based methods [60, 61, 63, 67, 70, 71, 73], DALL-E [48] and CogView [13], on MSCOCO, CUB-200 and Oxford-102 datasets.”
Improving Text-to-Image Synthesis Using Contrastive Learning (2021, methodology): “Recently, a great number of studies [1, 4, 6, 12, 13, 14, 17, 18, 19, 22, 23, 24, 25, 32, 33, 36, 42, 43, 44, 45, 46, 47, 48] present promising results on the text-to-image synthesis task, most of which make use of GANs as the backbone model.”
Generating images from caption and vice versa via CLIP-Guided Generative Latent Space Search (2021, methodology): “In the last years, some architectures like StackGAN++ (Zhang et al., 2018) and AlignDRAW (Mansimov et (Radford et al., 2021) consists of two encoders: one for images and another for texts.”
DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-To-Image Synthesis (2019, methodology): “StackGAN [30] and StackGAN++ [31] generate photo-realistic high-resolution images with two stages.”
AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks (2017, methodology): “Inception scores by state-of-the-art GAN models [20, 18, 36, 37, 16] and our AttnGAN on CUB and COCO test sets.”
Scaling Autoregressive Models for Content-Rich Text-to-Image Generation (2022, background): “Later works improved upon generation quality by introducing progressive refinement [68, 69] and using cross-modal attention mechanisms [47, 54].”
Object-Driven Text-To-Image Synthesis via Adversarial Training (2019, methodology): “We apply the patch-wise discriminator for every stage of the generation, following [33, 11], but only apply the objectwise discriminator at the final stage.”
AutoGAN: Neural Architecture Search for Generative Adversarial Networks (2019, background): “It has achieved great success in many generation and synthesis tasks, such as text-to-image translation [66, 65, 61, 48], image-to-image translation [22, 70, 63], and image enhancement [33, 31, 23].”
Transformers in Vision: A Survey (2021, methodology): “Prior text-to-image synthesis approaches [169, 225, 240, 241] are mostly based on GANs [71].”
Adversarial Latent Autoencoders (2020, background): “LAPGAN [5] and StackGAN [55, 56] train a stack of GANs organized in a multi-resolution pyramid to generate highresolution images.”
Paint by Word (2021, background): “The second approach is to train a conditional generator that explicitly takes a language embedding as input [35]; the image quality of this approach can be improved using multi-stage generators [47, 48, 50], and semantic consistency can be improved through careful design of architectures and trainin”
Controllable Text-to-Image Generation (2019, methodology): “For instance, as shown in the first two columns, when we modify the colour attributes, StackGAN++ changes the pose of the bird, and AttnGAN generates new background.”
Efficient Feature Transformations for Discriminative and Generative Continual Learning (2021, methodology): “We use the default StackGAN-v2 hyperparameter values for learning rate and the number of epochs.”
Semantics Disentangling for Text-To-Image Generation (2019, methodology): “Inspired by [40, 37, 14, 41], we adopt hierarchical stages from lowresolution to high-resolution for the photo-realistic image generation.”
MirrorGAN: Learning Text-To-Image Generation by Redescription (2019, methodology): “First, we compare MirrorGAN with state-of-the-art text-to-image methods [24, 25, 39, 40, 20, 35] using the Inception Score and R-precision score on both CUB and COCO datasets.”