GANs Trained by a Two Time-Scale Update Rule Converge to a Nash Equilibrium

By Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, Günter Klambauer, Sepp Hochreiter. 2017

506 citations recorded by the source corpus.

Generative Adversarial Networks (GANs) excel at creating realistic images with complex models for which maximum likelihood is infeasible. However, the convergence of GAN training has still not been proved. We propose a two time-scale update rule (TTUR) for training GANs with stochastic gradient descent on arbitrary GAN loss functions. TTUR has an individual learning rate for both the discriminator and the generator. Using the theory of stochastic approximation, we prove that the TTUR converges under mild assumptions to a stationary local Nash equilibrium. The convergence carries over to the popular Adam optimization, for which we prove that it follows the dynamics of a heavy ball with friction and thus prefers flat minima in the objective landscape. For the evaluation of the performance of GANs at image generation, we introduce the "Frechet Inception Distance" (FID) which captures the similarity of generated images to real ones better than the Inception Score. In experiments, TTUR improves learning for DCGANs and Improved Wasserstein GANs (WGAN-GP) outperforming conventional GAN training on CelebA, CIFAR-10, SVHN, LSUN Bedrooms, and the One Billion Word Benchmark.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark (2019, background): “Generative Evaluation Evaluation of generative quality is a difﬁcult goal with numerous proxy metrics, such as reconstruction error, FID (Heusel et al., 2017), IS (Sali-mans et al., 2016), precision-recall (Sajjadi et al., 2018), log likelihood (for auto-regressive models), or the ELBO (for VAEs).”
Are GANs Created Equal? A Large-Scale Study (2017, methodology): “Proposed by [10], FID provides an alternative approach.”
Large Scale GAN Training for High Fidelity Natural Image Synthesis (2018, background): “A variety of works have proposed heuristics for measuring the sample quality of models without tractable likelihoods (Salimans et al., 2016; Heusel et al., 2017; Bińkowski et al., 2018; Wu et al., 2017).”
Spectral Normalization for Generative Adversarial Networks (2018, methodology): “The algorithm based on the hinge loss also showed good performance when evaluated with inception score and FID.”
Denoising Diffusion Implicit Models (2020, methodology): “In Table 1, we report the quality of the generated samples with models trained on CIFAR10 and CelebA, as measured by Frechet Inception Distance (FID (Heusel et al., 2017)), where we vary the number of timesteps used to generate a sample (dim(τ)) and the stochasticity of the process (η).”
Self-Supervised GANs via Auxiliary Rotation Loss (2018, methodology): “Metrics To evaluate generated samples from different methods quantitatively, we use the Frechet Inception Distance (FID) [28].”
cGANs with Projection Discriminator (2018, background): “More precisely, FID (Heusel et al., 2017) measures the 2-Wasserstein distance between the two distributions q y and p y , and is given by F ( q y , p y ) = (cid:107) µ q y − µ p y 22 + trace (cid:0) C q y + C p y − 2( C q y C p y ) 1 / 2 , where { µ q y , C q y } , { µ p y , C p y } are…”
Object-Driven Text-To-Image Synthesis via Adversarial Training (2019, methodology): “We notice that the increment of batch size does boost the Inception score and R-precision, but does not improve FID.”
Demystifying MMD GANs (2018, background): “FID The Fréchet Inception Distance , proposed by Heusel et al. (2017), avoids some of the problems of Inception by measuring the similarity of the samples’ representations in the Inception architecture (at the pool3 layer, of dimension 2048 ) to those of samples from the target distribution.”
AutoGAN: Neural Architecture Search for Generative Adversarial Networks (2019, methodology): “On one hand, if we use a pre-ﬁxed D (or G ) and search only for G (or D ), it will easily incur imbalance between the powers of D or G [18, 2] (especially, at the early stage of NAS), resulting in slow updates or trivial learning.”
Counterfactuals uncover the modular structure of deep generative models (2018, neutral): “We further evaluated the relative quality of the counterfactual images with respect to the original generated images using the Frechet Inception Distance (FID) (Heusel et al., 2017) (Table 2 in the appendix), supporting that the hybridization procedure only mildly affects the image quality, in compa”
Memory Replay GANs: learning to generate images from new categories without forgetting (2018, background): “Note that FID is sensitive to both quality and diversity[8].”
The GAN Landscape: Losses, Architectures, Regularization, and Normalization (2018, methodology): “[12] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, Günter Klambauer, and Sepp Hochreiter.”
On Self Modulation for Generative Adversarial Networks (2018, background): “An alternative score, the Frechet Inception Distance (FID), requires no labeled data (Heusel et al., 2017).”
High-Fidelity Image Generation With Fewer Labels (2019, neutral): “…BigGAN, we do not apply orthogonal regularization as this was observed to only marginally improve sample quality (cf. Evaluation metrics We use the Fr´echet Inception Distance (FID) (Heusel et al., 2017) and Inception Score (Sal-imans et al., 2016) to evaluate the quality of the generated samples.”
The Mechanics of n-Player Differentiable Games (2018, methodology): “Heusel et al. (2017) propose a two-time scale methods to ﬁnd Nash.”
The Unusual Effectiveness of Averaging in GAN Training (2018, methodology): “At every 10k iterations, Inception and FID scores are evaluated, and we pick the best point.”
An empirical study on evaluation metrics of generative adversarial networks (2018, methodology): “…feature), FID models φ ( P r ) and φ ( P g ) as Gaussian random variables with empirical means µ r , µ g and empirical covariance C r , C g , and computes which is the Fréchet distance (or equivalently, the Wasserstein-2 distance) between the two Gaussian distributions (Heusel et al., 2017).”
Evaluation of Text Generation: A Survey (2020, background): “Fréchet Inception Distance Extending Inception score, Heusel et al. (2017) propose a new metric called Fréchet Inception Distance (FID) to score the similarity between generated images and real ones.”