Stabilizing GAN Training with Multiple Random Projections

By Behnam Neyshabur, Srinadh Bhojanapalli, Ayan Chakrabarti. 2017

85 citations recorded by the source corpus.

Training generative adversarial networks is unstable in high-dimensions as the true data distribution tends to be concentrated in a small fraction of the ambient space. The discriminator is then quickly able to classify nearly all generated samples as fake, leaving the generator without meaningful gradients and causing it to deteriorate after a point in training. In this work, we propose training a single generator simultaneously against an array of discriminators, each of which looks at a different random low-dimensional projection of the data. Individual discriminators, now provided with restricted views of the input, are unable to reject generated samples perfectly and continue to provide meaningful gradients to the generator throughout training. Meanwhile, the generator learns to produce samples consistent with the full data distribution to satisfy all discriminators simultaneously. We demonstrate the practical utility of this approach experimentally, and show that it is able to produce image samples with higher quality than traditional training with a single discriminator.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

In Optimization & Training

Citations in the corpus

Projected GANs Converge Faster (2021, background): “A different line of work considers a setup with multiple discriminators, applied to either the generated RGB image [8, 13] or low-dimensional projections thereof [1, 40].”
Generative Adversarial Network Training is a Continual Learning Problem (2018, background): “…several proposed models utilizing multiple generators (Hoang et al., 2018; Ghosh et al., 2018) or multiple discriminators (Durugkar et al., 2017; Neyshabur et al., 2017), while Bayesian GAN (Saatchi & Wilson, 2017) considered distributions on the parameters of both networks, but these all do…”
Evolutionary Generative Adversarial Networks (2018, background): “Multidiscriminator GAN frameworks [46], [47] are devised and utilized for providing stable gradients to the generator and further stabilizing the adversarial training process.”
Adversarial target-invariant representation learning for domain generalization (2019, background): “Namely, a random projection layer is introduced to the input of each domain discriminator, as in (Neyshabur et al., 2017; Albuquerque et al., 2019).”
HP-GAN: Probabilistic 3D Human Motion Prediction via GAN (2017, background): “Recent literature[1, 10, 25, 37, 29, 22] tries to improve GAN training and provide a theoretical guaranty for its convergence.”
Multi-objective training of Generative Adversarial Networks with multiple discriminators (2019, methodology): “The upper bound proven by Neyshabur et al. (2017) assumes that the marginals of the real and generated distributions are identical along all random projections.”
Generative Adversarial Networks (GANs) (2020, background): “Due to this, D can easily classify almost all generated samples as fake and does not provide meaningful gradients to D. Neyshabur, et al. [68] proposed a different approach to handle the instability in which G is trained against an array of Ds where each D handles different, randomly-chosen, low-dim”
Online Adaptative Curriculum Learning for GANs (2018, background): “…architectures (Huang et al. 2016; Karras et al. 2017), mutual information / entopy maximization (Belghazi et al. 2018), multiple discriminators (Neyshabur, Bhojanapalli, and Chakrabarti 2017; Juefei-Xu, Boddeti, and Savvides 2017), or multiple generators (Tolstikhin et al. 2017; Hoang et al.…”
Generative Modeling Using the Sliced Wasserstein Distance (2018, background): “They are known to produce sharp examples, but, among others due to the saddle-point formulation inherent to two-player games, optimization is ﬁnicky, as justiﬁed by the many papers addressing this topic [1, 9, 31, 26, 27, 22, 20, 25, 13, 11, 19].”
AmbientGAN: Generative models from lossy measurements (2018, background): “[Neyshabur et al. (2017)] explores an interesting connection of training stability with low dimensional projections of samples.”
Dropout-GAN: Learning from a Dynamic Ensemble of Discriminators (2018, background): “However, having weaker discriminators by training each one of them with fewer samples than G, is not necessarily bad since they are more likely to give positive feedback to G [10, 29].”