Gradient descent GAN optimization is locally stable

By Vaishnavh Nagarajan, J. Kolter. 2017

375 citations recorded by the source corpus.

Despite the growing prominence of generative adversarial networks (GANs), optimization in GANs is still a poorly understood topic. In this paper, we analyze the "gradient descent" form of GAN optimization i.e., the natural setting where we simultaneously take small gradient steps in both generator and discriminator parameters. We show that even though GAN optimization does not correspond to a convex-concave game (even for simple parameterizations), under proper conditions, equilibrium points of this optimization procedure are still \emph{locally asymptotically stable} for the traditional GAN formulation. On the other hand, we show that the recently proposed Wasserstein GAN can have non-convergent limit cycles near equilibrium. Motivated by this stability analysis, we propose an additional regularization term for gradient descent GAN updates, which \emph{is} able to guarantee local stability for both the WGAN and the traditional GAN, and also shows practical promise in speeding up convergence and addressing mode collapse.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Optimization & Training

Citations in the corpus

A Closer Look at the Optimization Landscapes of Generative Adversarial Networks (2019, background): “Recently, Nagarajan and Kolter (2017); Mescheder et al. (2018) show that a local analysis of the eigenvalues of the Jacobian of the game can provide guarantees on local stability properties.”
Negative Momentum for Improved Game Dynamics (2018, background): “Nagarajan and Kolter (2017) also analyze the local stability of GANs as an approximated continuous dynamical system.”
Which Training Methods for GANs do actually Converge? (2018, methodology): “…Gulrajani et al., 2017; Kodali et al., 2017; Sønderby et al., 2016; Roth et al., 2017) for GANs as well as gaining better theoretically understanding of their training dynamics (Arjovsky et al., 2017; Arjovsky & Bottou, 2017; Mescheder et al., 2017; Nagarajan & Kolter, 2017; Heusel et al., 2017).”
GANs Trained by a Two Time-Scale Update Rule Converge to a Nash Equilibrium (2017, methodology): “Recently the local stability of GANs and their convergence rates have been investigated [41].”
A Variational Inequality Perspective on Generative Adversarial Networks (2018, methodology): “The minimax formulation from Goodfellow et al. (2014) gives: min Note that as observed by Nagarajan and Kolter (2017), this objective is concave-concave, making it hard to optimize.”
The Mechanics of n-Player Differentiable Games (2018, background): “Nagarajan & Kolter (2017) prove convergence for some algorithms, but under very strong assumptions Mescheder (2018).”
Adversarial Latent Autoencoders (2020, methodology): “Following the more general formulation introduced in [39], the GAN learning problem entails finding the minimax with respect to the pair (G, D) (i.”
The Unusual Effectiveness of Averaging in GAN Training (2018, methodology): “Due to the complexity of GANs, local stability analysis is often used as a method for obtaining theoretical insights, see e.g. Nagarajan & Kolter (2017); Mescheder et al. (2018).”
Loss-Sensitive Generative Adversarial Networks on Lipschitz Densities (2017, background): “Finally, we note that the study of generalization performances is not directly related with the stability of training methods (Nagarajan and Kolter 2017; Yadav et al. 2017).”
PacGAN: The Power of Two Samples in Generative Adversarial Networks (2017, background): “Empirically, this observation seems to hold even with improved GAN optimization techniques, such as unrolled GANs [46], despite recent work showing that gradient-descent-based optimization of GANs is locally stable [50].”
Improving MMD-GAN Training with Repulsive Loss Function (2018, methodology): “On the other hand, we cite Theorem A.4 from Nagarajan & Kolter (2017): Lemma A.2 (Nagarajan & Kolter (2017)).”
Mitigating Information Leakage in Image Representations: A Maximum Entropy Approach (2019, background): “[16] both leveraged tools from non-linear systems theory [9] to analyze the convergence properties of simultaneous gradient descent in the context of GANs.”
Interaction Matters: A Note on Non-asymptotic Local Convergence of Generative Adversarial Networks (2018, background): “The local exponential stability in the inﬁnitesimal/asymptotic case when η → 0 has already been studied in a nice paper Nagarajan and Kolter [2017] (Theorem 3.1 therein) by showing the Jacobian matrix of a particular form of GAN objective is Hurwitz (has all strictly negative eigenvalues).”
Multivariate Time Series Imputation with Generative Adversarial Networks (2018, background): “Because of the mode collapse problem [3], the traditional GAN is hard to train [20, 32, 37].”
Making Method of Moments Great Again? -- How can GANs learn the target distribution (2020, background): “2 Related works Generative Adversarial Networks The works [62, 41, 58, 28, 29, 37, 54, 61, 55] considered either local convergence property of the GAN training, or the global convergence property when the objective is convex-concave However, there is no clear evidence of how the local property can b”