Stacked Generative Adversarial Networks

By Xun Huang, Yixuan Li, Omid Poursaeed, John E. Hopcroft, Serge J. Belongie. 2016

470 citations recorded by the source corpus.

In this paper, we propose a novel generative model named Stacked Generative Adversarial Networks (SGAN), which is trained to invert the hierarchical representations of a bottom-up discriminative network. Our model consists of a top-down stack of GANs, each learned to generate lower-level representations conditioned on higher-level representations. A representation discriminator is introduced at each feature hierarchy to encourage the representation manifold of the generator to align with that of the bottom-up discriminative network, leveraging the powerful discriminative representations to guide the generative model. In addition, we introduce a conditional loss that encourages the use of conditional information from the layer above, and a novel entropy loss that maximizes a variational lower bound on the conditional entropy of generator outputs. We first train each stack independently, and then train the whole model end-to-end. Unlike the original GAN that uses a single noise vector to represent all the variations, our SGAN decomposes variations into multiple levels and gradually resolves uncertainties in the top-down generative process. Based on visual inspection, Inception scores and visual Turing test, we demonstrate that SGAN is able to generate images of much higher quality than GANs without stacking.

Subfields: Biophysics, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biophysics

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Progressive Growing of GANs for Improved Quality, Stability, and Variation (2017, background): “16) who use one generator and multiple discriminators concurrently, and Ghosh et al. (2017) who do the opposite with multiple generators and one discriminator. Hierarchical GANs (Denton et al., 2015; Huang et al., 2016; Zhang et al., 2017) deﬁne a generator and discriminator for each level of an ima”
Projected GANs Converge Faster (2021, neutral): “Pretrained models can be used as a guiding mechanism to disentangle causal generative factors [54], for text-driven image manipulation [44], matching the generator activations to inverted classifiers [19, 56], or to generate images via gradient ascent in the latent space of a generator [41].”
Improved Training of Wasserstein GANs (2017, neutral): “10 SGAN [11] 8 .”
High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs (2017, background): “Various coarse-to-fine schemes [4] have been proposed [9,19,26,55] to synthesize larger images (e.”
StarGAN: Unified Generative Adversarial Networks for Multi-domain Image-to-Image Translation (2017, background): “Generative adversarial networks (GANs) [3] have shown remarkable results in various computer vision tasks such as image generation [1, 6, 23, 31], image translation [7, 8, 32], super-resolution imaging [13], and face image synthesis [9, 15, 25, 30].”
Generalization and Equilibrium in Generative Adversarial Nets (GANs) (2017, methodology): “…method on image generation tasks: MNIST dataset [LeCun et al., 1998] of hand-written digits and the CelebA [Liu et al., 2015] dataset Liu, 2016] 6.35 Improved GAN [Salimans et al., 2016] 8.09 ± 0.07 AC-GAN [Odena et al., 2016] 8.25 ± 0.07 S-GAN (best variant in [Huang et al., 2017 of human faces.”
StackGAN: Text to Photo-Realistic Image Synthesis with Stacked Generative Adversarial Networks (2016, methodology): “[10] also showed that they can generate better images by stacking several GANs to reconstruct the multi-level representations of a pre-trained discriminative model.”
Least Squares Generative Adversarial Networks (2016, background): “Method Inception Score DCGAN (reported in [10]) 6.”
Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization (2017, background): “Various improvements to the GAN framework have been proposed, such as conditional generation [43, 23], multistage processing [9, 20], and better training objectives [46, 1].”
StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networks (2017, background): “[16] added several GANs to reconstruct the multi-level representations of a pre-trained discriminative model.”
Demystifying MMD GANs (2018, methodology): “This class of models has recently led to many impressive examples of image generation (e.g. Huang et al., 2017a;b; Jin et al., 2017; Zhu et al., 2017).”
Multimodal Unsupervised Image-to-Image Translation (2018, positive): “The style reconstruction loss Li recon is reminiscent of the latent reconstruction loss used in the prior works [11, 31, 44, 57].”
Do GANs actually learn the distribution? An empirical study (2017, background): “Indeed, when we test a DCGAN (even the best variant with 7.16 Inception Score [12]), the pairs returned are mostly blobs.”
Video-to-Video Synthesis (2018, background): “Careful designs of network architectures and objective functions are essential to achieve good performance as shown in the literature [14,21,30,37,49,51,55,73,80].”
Wasserstein Introspective Neural Networks (2017, neutral): “07 DCGAN [36] (in [18]) 6 .”
Semantics Disentangling for Text-To-Image Generation (2019, background): “A set of constraints are proposed in previous works [28, 16, 26, 9, 36] to improve the training process of GANs, e.”
Improving MMD-GAN Training with Repulsive Loss Function (2018, background): “Using a pre-trained classifier may also help produce vivid image samples (Huang et al. (2017)) as the learned representations of the real samples in the hidden layers of the classifier tend to be well separated/organized and may produce more meaningful gradients to the generator.”
SinGAN: Learning a Generative Model From a Single Natural Image (2019, background): “[28, 52, 29, 52, 13, 24]), we are the first explore it for internal learning from a single image.”
Photographic Text-to-Image Synthesis with a Hierarchically-Nested Adversarial Network (2018, methodology): “There are substantial existing methods investigating the better usage of GANs for different applications, such as image synthesis [32, 38], (unpaired) pixelto-pixel translation [14, 51], medical applications [4, 49], etc [20, 12, 52, 45].”
Pros and Cons of GAN Evaluation Measures (2018, neutral): “Similar approaches have been followed in [36, 99, 94, 96, 100, 82, 18, 52, 56].”