Generalization and Equilibrium in Generative Adversarial Nets (GANs)

By Sanjeev Arora, Rong Ge, Yingyu Liang, Tengyu Ma, Yi Zhang. 2017

727 citations recorded by the source corpus.

We show that training of generative adversarial network (GAN) may not have good generalization properties; e.g., training may appear successful but the trained distribution may be far from target distribution in standard metrics. However, generalization does occur for a weaker metric called neural net distance. It is also shown that an approximate pure equilibrium exists in the discriminator/generator game for a special class of generators with natural training objectives when generator capacity and training set sizes are moderate. This existence of equilibrium inspires MIX+GAN protocol, which can be combined with any existing GAN training, and empirically shown to improve some of them.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Generative Models

Citations in the corpus

Are GANs Created Equal? A Large-Scale Study (2017, background): “There are several ongoing challenges in the study of GANs, including their convergence and generalization properties [2, 19], and optimization stability [24, 1].”
Imperfect ImaGANation: Implications of GANs Exacerbating Biases on Facial Data Augmentation and Snapchat Selfie Lenses (2020, background): “In reality, GAN-generated distributions are not nearly as diverse as their training distributions (Arora et al., 2018; 2017) and the support (i.e. possible feature combinations of the generated data) is only representative of a small subset of what one would expect to see when sampling data from…”
Gradient descent GAN optimization is locally stable (2017, background): “Second, the analysis here does not show the equilibrium points necessarily exist, but only illustrates convergence if there do exist points that satisfy certain criteria: the existence question has been addressed by previous work [Arora et al., 2017], but much more analysis remains to be done here.”
The Numerics of GANs (2017, background): “Therefore, some authors call these divergence functions neural network divergences [5].”
Demystifying MMD GANs (2018, methodology): “…although Goodfellow et al. (2014) motivated GANs as estimating the Jensen-Shannon divergence, they can also be viewed as minimizing the IPM deﬁned by the classiﬁer family (Arora et al., 2017; Liu et al., 2017), thus motivating applying the gradient penalty to original GANs (Fedus et al., 2018).”
BEGAN: Boundary Equilibrium Generative Adversarial Networks (2017, methodology): “62 ALI [5] 5.34 Improved GANs [16] 4.36 MIX + WGAN [2] 4.04 Table 1: Inception scores (higher is better) To measure quality and diversity numerically, we computed the inception score [16] on CIFAR-10 images.”
GANs Trained by a Two Time-Scale Update Rule Converge to a Nash Equilibrium (2017, neutral): “g and unrealistic assumptions or a restriction to a linear discriminator. Recent convergence proofs for GANs hold for expectations over training samples or for the number of examples going to inﬁnity [37, 45, 40, 4], thus do not consider mini-batch learning which leads to a stochastic gradient [57,2”
A Variational Inequality Perspective on Generative Adversarial Networks (2018, background): “, 2017), before presenting alternative viewpoints in terms of game theory (Arora et al., 2017) later in this section.”
Catastrophic forgetting and mode collapse in GANs (2020, background): “In this paper, we focus on the continual learning problem in the discriminator as many prior works have showed that the quality of a GAN mainly depends on its discriminator [16, 17, 18].”
Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samples (2017, background): “One can expect that samples from the original GAN can be also useful for improving the detection performance since it may have bad generalization properties [2] and generate a few samples on the low-density boundary as like the proposed GAN.”
MMD GAN: Towards Deeper Understanding of Moment Matching Network (2017, background): “In practice, [22] show neural networks does not have enough capacity to approximate Wasserstein distance.”
Do GANs actually learn the distribution? An empirical study (2017, methodology): “The Arora et al. scenario could involve the trained distribution having small support, and yet all its samples could be completely disjoint from the training samples.”
Is Generator Conditioning Causally Related to GAN Performance? (2018, background): “There have also been many efforts to better understand their training procedure, and in particular to understand various pathologies that seem to plague that training procedure (Metz et al., 2016; Arora et al., 2017; Heusel et al., 2017; Nagarajan & Kolter, 2017; Arjovsky 1Google Brain.”
LOGAN: Latent Optimisation for Generative Adversarial Networks (2019, neutral): “Our analysis can be generalised to other losses as in previous theoretical work (e.g., Arora et al. 2017).”
An empirical study on evaluation metrics of generative adversarial networks (2018, background): “However, it has a high sample complexity, a fact that has also been observed by (Arora et al., 2017).”
An Online Learning Approach to Generative Adversarial Networks (2017, methodology): “The closest work related to our approach is Arora et al. (2017) that showed the existence of an approximate mixed equilibrium with certain generalization properties; yet without providing a constructive way to ﬁnd such equilibria.”
Multi-agent Diverse Generative Adversarial Networks (2017, background): “In terms of employing multiple generators, our work is closest to [4, 20, 12].”