Approximation and Convergence Properties of Generative Adversarial Learning

By Shuang Liu, Olivier Bousquet, Kamalika Chaudhuri. 2017

145 citations recorded by the source corpus.

Generative adversarial networks (GAN) approximate a target data distribution by jointly optimizing an objective function through a "two-player game" between a generator and a discriminator. Despite their empirical success, however, two very basic questions on how well they can approximate the target distribution remain unanswered. First, it is not known how restricting the discriminator family affects the approximation quality. Second, while a number of different objective functions have been proposed, we do not understand when convergence to the global minima of the objective function leads to convergence to the target distribution under various notions of distributional convergence. In this paper, we address these questions in a broad and unified setting by defining a notion of adversarial divergences that includes a number of recently proposed objective functions. We show that if the objective function is an adversarial divergence with some additional conditions, then using a restricted discriminator family has a moment-matching effect. Additionally, we show that for objective functions that are strict adversarial divergences, convergence in the objective function implies weak convergence, thus generalizing previous results.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Improved Training of Wasserstein GANs (2017, background): “Although GANs do not literally minimize these divergences [16], in practice this means a discriminator might quickly learn to reject all samples that don’t lie on V T n (sequences of one-hot vectors) and give meaningless gradients to the generator.”
Demystifying MMD GANs (2018, methodology): “(2014) motivated GANs as estimating the Jensen-Shannon divergence, they can also be viewed as minimizing the IPM defined by the classifier family (Arora et al., 2017; Liu et al., 2017), thus motivating applying the gradient penalty to original GANs (Fedus et al.”
GANs Trained by a Two Time-Scale Update Rule Converge to a Nash Equilibrium (2017, neutral): “g and unrealistic assumptions or a restriction to a linear discriminator. Recent convergence proofs for GANs hold for expectations over training samples or for the number of examples going to inﬁnity [37, 45, 40, 4], thus do not consider mini-batch learning which leads to a stochastic gradient [57,2”
A Variational Inequality Perspective on Generative Adversarial Networks (2018, methodology): “We start with the generative adversarial net formulation understood as divergence minimization (Liu et al., 2017), before presenting alternative viewpoints in terms of game theory (Arora et al.”
PacGAN: The Power of Two Samples in Generative Adversarial Networks (2017, background): “, the generator does not learn the true data distribution; this phenomenon is of significant interest [2, 3, 18, 1, 2].”
Improving MMD-GAN Training with Repulsive Loss Function (2018, background): “Weak metrics are desirable as they provide more information on adjusting the model to fit the data distribution (Liu et al. (2017)).”
Interaction Matters: A Note on Non-asymptotic Local Convergence of Generative Adversarial Networks (2018, background): “Recently, several attempts have been made to understand whether GANs learn the target distribution in the statistical sense [Liu et al., 2017, Arora and Zhang, 2017, Liang, 2017, Arora et al., 2017].”
Parametric Adversarial Divergences are Good Task Losses for Generative Modeling (2017, methodology): “, 2017) and more intuitive (Liu et al., 2017) than stronger divergences.”
On the Discrimination-Generalization Tradeoff in GANs (2017, background): “Our results on neural φ -divergence can in general extended to the more uniﬁed framework of Liu et al. (2017) in which divergences of form max f E ( x,y ) ∼ µ ⊗ ν [ f ( x, y )] are studied.”
Learning Implicit Generative Models with the Method of Learned Moments (2018, background): “Recent work (Liu et al., 2017) has shown that in practical settings, many GAN objectives are better expressed as gen-”
On How Well Generative Adversarial Networks Learn Densities: Nonparametric and Parametric Results (2018, background): “We refer the readers to Liu et al. (2017) for other more general formulations of GANs.”
Bridging the Gap Between $f$-GANs and Wasserstein GANs (2019, methodology): “Several works (Liu et al., 2017; Farnia & Tse, 2018) considered restricting function classes directly over the f -GAN objective; Husain et al.”
Generalized Energy Based Models (2020, background): “The converse is a direct consequence of (Liu et al., 2017, Theorem 10) since by assumptionX is compact.”
Multivariate Time Series Imputation with Generative Adversarial Networks (2018, background): “Because of the mode collapse problem [3], the traditional GAN is hard to train [20, 32, 37].”
Hausdorff GAN: Improving GAN Generation Quality With Hausdorff Metric (2021, background): “Thus, it can provide more information about original data in order to adjust the model to ﬁt original data distribution [32].”
GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium (2017, background): “35, 2], thus do not consider mini-batch learning which leads to To get a more concrete sense of the two time-scale algorithms based on primal decomposition, we consider the following NUM problem: analyzed using stochastic approximation.”
Geometrical Insights for Implicit Generative Modeling (2017, methodology): “On the other hand, this toolbox essentially works by restricting the familyQ in ways that can change the meaning of the comparison criteria itself [5, 34].”