The Numerics of GANs

By Lars M. Mescheder, Sebastian Nowozin, Andreas Geiger. 2017

485 citations recorded by the source corpus.

In this paper, we analyze the numerics of common algorithms for training Generative Adversarial Networks (GANs). Using the formalism of smooth two-player games we analyze the associated gradient vector field of GAN training objectives. Our findings suggest that the convergence of current algorithms suffers due to two factors: i) presence of eigenvalues of the Jacobian of the gradient vector field with zero real-part, and ii) eigenvalues with big imaginary part. Using these findings, we design a new algorithm that overcomes some of these limitations and has better convergence properties. Experimentally, we demonstrate its superiority on training common GAN architectures and show convergence on GAN architectures that are known to be notoriously hard to train.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Generative Models

Citations in the corpus

Are GANs Created Equal? A Large-Scale Study (2017, background): “There are several ongoing challenges in the study of GANs, including their convergence and generalization properties [2, 19], and optimization stability [24, 1].”
Stochastic Hamiltonian Gradient Methods for Smooth Games (2020, methodology): “Existing practical variants of SHGD use biased estimators (Mescheder et al., 2017).”
A Tight and Unified Analysis of Extragradient for a Whole Spectrum of Differentiable Games (2019, methodology): “For instance, following Mescheder et al. (2017); Gidel et al. (2019b), the standard formulation of GANs from Goodfellow et al. (2014) can be cast as a two-player zero-sum game.”
A Closer Look at the Optimization Landscapes of Generative Adversarial Networks (2019, methodology): “In the context of GANs, Mescheder et al. (2017) introduced a gradient penalty and use the eigenvalues of the Jacobian of the game vector ﬁeld to show its beneﬁts in terms of stability.”
Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Rates (2019, methodology): “Furthermore, the interpolation property can be used to improve the convergence for a bilinear saddle-point problem [24, 25, 54, 85].”
Reducing Noise in GAN Training with Variance Reduced Extragradient (2019, neutral): “The gradient method does not converge for some convex-concave examples (Mescheder et al., 2017; Gidel et al., 2019a).”
Negative Momentum for Improved Game Dynamics (2018, methodology): “Optimization From an optimization point of view, a lot of work has been done in the context of understanding momentum and its variants (Polyak, 1964; Qian, 1999; Nesterov, 2013; Sutskever et al., 2013). Some recent studies have emphasized on the importance of momentum tuning in deep learning such as”
Gradient descent GAN optimization is locally stable (2017, methodology): “Of particular similarity to the methodology we propose here are the works by Roth et al. [2017] and Mescheder et al. [2017].”
Which Training Methods for GANs do actually Converge? (2018, methodology): “For simultaneous gradient descent linear convergence can be achieved if and only if all eigenvalues of the Jacobian of the gradient vector ﬁeld v ( θ, ψ ) have negative real part (Mescheder et al., 2017).”
GANs Trained by a Two Time-Scale Update Rule Converge to a Nash Equilibrium (2017, background): “oofs for GANs hold for expectations over training samples or for the number of examples going to inﬁnity [37, 45, 40, 4], thus do not consider mini-batch learning which leads to a stochastic gradient [57,25,42,38]. Recently actor-critic learning has been analyzed using stochastic approximation. Pras”
A Variational Inequality Perspective on Generative Adversarial Networks (2018, background): “Li et al. (2017) suggest to dualize the GAN objective to reformulate it as a maximization problem and Mescheder et al. (2017) propose to add the norm of the gradient in the objective to get a better signal.”
Catastrophic forgetting and mode collapse in GANs (2020, background): “Prior works on the convergence of GANs usually consider the convergence in parameter space [9, 11, 12, 13].”
Stabilizing Training of Generative Adversarial Networks through Regularization (2017, methodology): “In addition to the deep convolutional GAN (DCGAN) of [26], we trained several common architectures that are known to be hard to train [4, 26, 19], therefore allowing us to establish a comparison to the concurrently proposed gradientpenalty regularizer for Wasserstein GANs [10].”
The Mechanics of n-Player Differentiable Games (2018, methodology): “We compare SGA with simultaneous gradient descent, optimistic mirror descent (Daskalakis et al., 2018) and consensus optimization (Mescheder et al., 2017) in basic settings.”
The Unusual Effectiveness of Averaging in GAN Training (2018, methodology): “There might be multiple reasons for this cycling behavior: (i) eigenvalues of the Jacobian of the gradient vector ﬁeld have zero real part and a large imaginary part (Mescheder et al., 2017).”
LOGAN: Latent Optimisation for Generative Adversarial Networks (2019, neutral): “3) for our analysis (see also Mescheder et al. 2017; 2018).”
PacGAN: The Power of Two Samples in Generative Adversarial Networks (2017, background): “Recently, other innovative choices for regularizations have been proposed, including weight clipping [1], gradient regularization [20, 45, 50], Tikhonov regularizer for training-with-noise [58], adding noise to the gradient [21], and spectral-norm regularization [49].”
Mitigating Information Leakage in Image Representations: A Maximum Entropy Approach (2019, methodology): “The parameters of the models in ARL are typically optimized through stochastic gradient descent, either jointly [3, 15] or alternatively [4].”