StackGAN: Text to Photo-Realistic Image Synthesis with Stacked Generative Adversarial Networks
By Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, Dimitris N. Metaxas. 2016
2,951 citations recorded by the source corpus.
Synthesizing high-quality images from text descriptions is a challenging problem in computer vision and has many practical applications. Samples generated by existing textto- image approaches can roughly reflect the meaning of the given descriptions, but they fail to contain necessary details and vivid object parts. In this paper, we propose Stacked Generative Adversarial Networks (StackGAN) to generate 256.256 photo-realistic images conditioned on text descriptions. We decompose the hard problem into more manageable sub-problems through a sketch-refinement process. The Stage-I GAN sketches the primitive shape and colors of the object based on the given text description, yielding Stage-I low-resolution images. The Stage-II GAN takes Stage-I results and text descriptions as inputs, and generates high-resolution images with photo-realistic details. It is able to rectify defects in Stage-I results and add compelling details with the refinement process. To improve the diversity of the synthesized images and stabilize the training of the conditional-GAN, we introduce a novel Conditioning Augmentation technique that encourages smoothness in the latent conditioning manifold. Extensive experiments and comparisons with state-of-the-arts on benchmark datasets demonstrate that the proposed method achieves significant improvements on generating photo-realistic images conditioned on text descriptions.
Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Generative Models
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Graphics and Computer-Aided Design
- Jun-Yan Zhu
- Eli Shechtman
- Richard Zhang
- Alexei A. Efros
- Tianfu Wu
- Björn Ommer
- Robin Rombach
- Andrea Vedaldi
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Ming-Hsuan Yang
- Trevor Darrell
- Xiaogang Wang
- Jitendra Malik
- Yi Yang
- Bernt Schiele
In Generative Models
- Jun-Yan Zhu
- Richard Zhang
- Eli Shechtman
- Tianfu Wu
- Alexei A. Efros
- Stefano Ermon
- Daniel Cohen-Or
- Bolei Zhou
Citations in the corpus
- Image Transformer (2018, background): “…to the often blurry images generated by networks trained with likelihood-based losses, generative adversarial networks (GANs) have been shown to produce sharper images with realistic high-frequency detail in generation and image super-resolution tasks (Zhang et al., 2016; Ledig et al., 2016).”
- Are GANs Created Equal? A Large-Scale Study (2017, background): “Generative adversarial networks (GAN) are a powerful subclass of generative models and were successfully applied to image generation and editing, semi-supervised learning, and domain adaptation [22, 27].”
- Zero-Shot Text-to-Image Generation (2021, methodology): “These include improving the generative model architecture with modifications like multi-scale generators (Zhang et al., 2017; 2018), integrating attention and auxiliary losses (Xu et al., 2018), and leveraging additional sources of conditioning information beyond just text (Reed et al., 2016a; Li…”
- Progressive Growing of GANs for Improved Quality, Stability, and Variation (2017, background): “While the quality of our results is generally high compared to earlier work on GANs, and the training is stable in large resolutions, there is a long way to true photorealism.”
- CogView: Mastering Text-to-Image Generation via Transformers (2021, methodology): “StackGAN [54] decomposed the generation into a sketch-refinement process.”
- Diffusion Models Beat GANs on Image Synthesis (2021, background): “These models can be used in a variety of ways, such as generating images from text prompts [78, 56] or learning useful feature representations [18, 10].”
- Learning Layout and Style Reconfigurable GANs for Controllable Image Synthesis (2020, methodology): “Spatial layouts are usually used as intermediate representations for other conditional image synthesis tasks such as text-to-image [17], [21] and scene-graph-toimage [18], [19].”
- Object-Centric Image Generation from Layouts (2020, methodology): “2019; Brock, Donahue, and Simonyan 2019), captions (Reed et al. 2016; Zhang et al. 2017; Xu et al. 2018b; Li et al. 2019a; Yin et al. 2019), conversations (Sharma et al.”
- Semantic Image Synthesis With Spatially-Adaptive Normalization (2019, methodology): “Recent methods directly learn the mapping using neural networks [3, 6, 22, 47, 48, 54, 55, 56].”
- Vector Quantized Diffusion Model for Text-to-Image Synthesis (2021, methodology): “Compared with previous GAN-based text-to-image methods [62, 65, 66, 68], our method can handle more complex scenes and the synthesized image quality is improved by a large margin.”
- Improving Text-to-Image Synthesis Using Contrastive Learning (2021, methodology): “Recently, a great number of studies [1, 4, 6, 12, 13, 14, 17, 18, 19, 22, 23, 24, 25, 32, 33, 36, 42, 43, 44, 45, 46, 47, 48] present promising results on the text-to-image synthesis task, most of which make use of GANs as the backbone model.”
- DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-To-Image Synthesis (2019, methodology): “StackGAN [30] and StackGAN++ [31] generate photo-realistic high-resolution images with two stages.”
- AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks (2017, methodology): “We preprocess the CUB dataset according to the method in [36].”
- High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs (2017, background): “Various coarse-to-fine schemes [4] have been proposed [9,19,26,55] to synthesize larger images (e.”
- StarGAN: Unified Generative Adversarial Networks for Multi-domain Image-to-Image Translation (2017, background): “Other recent approaches focused on generating particular images highly relevant to a given text description [24, 29].”
- Unsupervised Image-to-Image Translation Networks (2017, background): “(Zhang et al., 2016) stacks two generators to progressively render realistic images.”