Do GANs actually learn the distribution? An empirical study

By Sanjeev Arora, Yi Zhang. 2017

199 citations recorded by the source corpus.

Do GANS (Generative Adversarial Nets) actually learn the target distribution? The foundational paper of (Goodfellow et al 2014) suggested they do, if they were given sufficiently large deep nets, sample size, and computation time. A recent theoretical analysis in Arora et al (to appear at ICML 2017) raised doubts whether the same holds when discriminator has finite size. It showed that the training objective can approach its optimum value even if the generated distribution has very low support ---in other words, the training objective is unable to prevent mode collapse. The current note reports experiments suggesting that such problems are not merely theoretical. It presents empirical evidence that well-known GANs approaches do learn distributions of fairly low support, and thus presumably are not learning the target distribution. The main technical contribution is a new proposed test, based upon the famous birthday paradox, for estimating the support size of the generated distribution.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Generating High Fidelity Images with Subscale Pixel Networks and Multidimensional Upscaling (2018, methodology): “On the other hand, as opposed to for example adversarial methods (Arora & Zhang, 2017), MLE forces the model to support the entire empirical distribution.”
Progressive Growing of GANs for Improved Quality, Stability, and Variation (2017, background): “…have been suggested for measuring it, including inception score (Salimans et al., 2016), multi-scale structural similarity (MS-SSIM) (Odena et al., 2017; Wang et al., 2003), birthday paradox (Arora & Zhang, 2017), and explicit tests for the number of discrete modes discovered (Metz et al., 2016).”
On Buggy Resizing Libraries and Surprising Subtleties in FID Calculation (2021, background): “One idea is to conduct user studies on cloud-sourcing platforms for either assessing the samples’ image quality [17, 64, 82] or identifying duplicate images [2].”
Improved Precision and Recall Metric for Assessing Generative Models (2019, background): “While metrics have been proposed to estimate the degree of variation, these have not seen widespread use as they are subjective [1], domain specific [19], or not reliable enough [22].”
Coulomb GANs: Provably Optimal Nash Equilibria via Potential Fields (2017, methodology): “To further investigate how much variation the samples generated by the Coulomb GAN really contains, we followed the advice of Arora and Zhang [3] to estimate the support size of the generator’s distribution by checking how large a sample from the generator must be before we start generating…”
Generalization and Equilibrium in Generative Adversarial Nets (GANs) (2017, background): “This is empirically veriﬁed in a follow-14 up work[Arora and Zhang, 2017].”
Demystifying MMD GANs (2018, methodology): “We also considered the diagnostic test of Arora & Zhang (2017), which estimates the approximate number of “distinct” images produced by a GAN.”
Generating Natural Adversarial Examples (2017, methodology): “Given that there are some concerns about whether GANs actually learn the distribution (Arora & Zhang, 2017), it is worth noting that we can also incorporate other generative models such as Variational Auto-Encoders (VAEs) (Kingma & Welling, 2014) into our framework, as used in Hu et al. (2017) to…”
Is Generator Conditioning Causally Related to GAN Performance? (2018, methodology): “There have also been many efforts to better understand their training procedure, and in particular to understand various pathologies that seem to plague that training procedure (Metz et al., 2016; Arora et al., 2017; Heusel et al., 2017; Nagarajan & Kolter, 2017; Arjovsky 1Google Brain.”
Discriminator Rejection Sampling (2018, methodology): “In particular, GANs as presently trained don’t learn to reproduce the target distribution (Arora & Zhang, 2017).”
PacGAN: The Power of Two Samples in Generative Adversarial Networks (2017, background): “Mode collapse is a phenomenon commonly reported in the GAN literature [17], [37], [38], [39], [40], which can refer to two distinct concepts: (i) the generative model loses some modes that are present in the samples of the target distribution.”
Pros and Cons of GAN Evaluation Measures (2018, methodology): “Arora and Zhang [27] proposed to use the birthday paradox7 test to evaluate GANs as follows: (a) Pick a sample of size S from the generated distribution (b) Use an automated measure of image similarity to flag the k (e.g. k = 20) most similar pairs in the sample (c) Visually inspect the flagged pair”
Flow-GAN: Combining Maximum Likelihood and Adversarial Learning in Generative Models (2017, methodology): “In prior work, mode collapse is detected based on visual inspection or heuristic techniques (Goodfellow 2016; Arora and Zhang 2017).”
Bias and Generalization in Deep Generative Models: An Empirical Study (2018, neutral): “dimensionality [3], suggesting extremely high data requirements.”
Interaction Matters: A Note on Non-asymptotic Local Convergence of Generative Adversarial Networks (2018, background): “Recently, several attempts have been made to understand whether GANs learn the target distribution in the statistical sense [Liu et al., 2017, Arora and Zhang, 2017, Liang, 2017, Arora et al., 2017].”
Conditional Adversarial Domain Adaptation (2017, background): “Namely, even if the discriminator is fully confused, we have no theoretical guarantee that two different distributions are made identical (Arora et al., 2017; Arora & Zhang, 2017).”
Deep generative models for T cell receptor protein sequences (2019, background): “Such behavior is a persistent concern for deep generative models ( Arora et al., 2017 ; Arora and Zhang, 2017 ).”
Non-Adversarial Image Synthesis With Generative Latent Nearest Neighbors (2018, unknown): “The birthday paradox can be used to measure the extent of mode dropping [2]: The number of modes modeled by a generator can be estimated by generating a fixed number of images and counting the number of repeated images.”
On Aliased Resizing and Surprising Subtleties in GAN Evaluation (2022, background): “One idea is to conduct user studies on cloud-sourcing platforms for either assessing the samples’ image quality [13, 59, 76] or identifying duplicate images [1].”