Which Training Methods for GANs do actually Converge?

By Lars M. Mescheder, Andreas Geiger, Sebastian Nowozin. 2018

1,690 citations recorded by the source corpus.

Recent work has shown local convergence of GAN training for absolutely continuous data and generator distributions. In this paper, we show that the requirement of absolute continuity is necessary: we describe a simple yet prototypical counterexample showing that in the more realistic case of distributions that are not absolutely continuous, unregularized GAN training is not always convergent. Furthermore, we discuss regularization strategies that were recently proposed to stabilize GAN training. Our analysis shows that GAN training with instance noise or zero-centered gradient penalties converges. On the other hand, we show that Wasserstein-GANs and WGAN-GP with a finite number of discriminator updates per generator update do not always converge to the equilibrium point. We discuss these results, leading us to a new explanation for the stability problems of GAN training. Based on our analysis, we extend our convergence results to more general GANs and prove local convergence for simplified gradient penalties even if the generator and data distribution lie on lower dimensional manifolds. We find these penalties to work well in practice and use them to learn high-resolution generative image models for a variety of datasets with little hyperparameter tuning.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Training Generative Adversarial Networks with Limited Data (2020, methodology): “We kept most of the details unchanged, including network architectures [21], weight demodulation [21], path length regularization [21], lazy regularization [21], style mixing regularization [20], bilinear filtering in all up/downsampling layers [20], equalized learning rate for all trainable paramet”
A Style-Based Generator Architecture for Generative Adversarial Networks (2018, methodology): “We do not modify the discriminator or the loss function in any way, and our work is thus orthogonal to the ongoing discussion about GAN loss functions, regularization, and hyper-parameters [23, 41, 4, 37, 40, 33].”
Large Scale GAN Training for High Fidelity Natural Image Synthesis (2018, methodology): “For evaluation, we employ moving averages of G ’s weights following Karras et al. (2018); Mescheder et al. (2018), with a decay of 0 .”
Generative Adversarial Networks (2021, neutral): “SNGAN achieves an extraordinary advance on ImageNet, and better or equal quality on CIFAR-10 and STL-10, compared to the previous training stabilization techniques that include weight clipping [4], gradient penalty [74, 43], batch normalization [26], weight normalization [63], layer normalization [5”
Projected GANs Converge Faster (2021, background): “While discriminator regularization techniques help to balance the adversarial game [31], standard regularization methods like gradient penalties [36] are susceptible to hyperparameter choices [26] and can lead to a substantial decrease in performance [4].”
Large Scale Image Completion via Co-Modulated Generative Adversarial Networks (2021, methodology): “…M (8-layer MLP), layer-wise noise injection, style mixing regularization (with a probability of 0.5 instead), non-saturating logistic loss (Goodfellow et al., 2014) with R1 regularization (Mescheder et al., 2018) of γ = 10, and the Adam optimizer (Kingma & Ba, 2014) with a learning rate of 0.002.”
Improved Precision and Recall Metric for Assessing Generative Models (2019, background): “We observe that R1 regularization [18] has a similar effect: reducing the γ parameter by 100× shifts the balance even further (C).”
Semantic Image Synthesis With Spatially-Adaptive Normalization (2019, methodology): “We test several ResNet-based discriminators used in recent unconditional GANs [1, 36, 39] but observe similar results at the cost of a higher GPU memory requirement.”
Self-Supervised GANs via Auxiliary Rotation Loss (2018, methodology): “Consequently, training may exhibit instability, divergence, cyclic behavior, or mode collapse [3].”
Learning deep representations by mutual information estimation and maximization (2018, methodology): “We used the contractive penalty (found in Mescheder et al. (2018) but ﬁrst introduced in contractive autoencoders (Rifai et al., 2011)) on the encoder, gradient clipping on the discriminators, and no regularization on the generator.”
Aligning Language Models with Preferences through f-divergence Minimization (2023, methodology): “1, we show additional metrics on generated sentences, which show low diversity but high quality for RKL-DPG, compared to other f -DPGs, suggesting it captures a subset of the target distribution (“mode collapse”), as commonly observed in other generative models (Huszar, 2015; Che et al., 2017; Mesch”
A Closer Look at the Optimization Landscapes of Generative Adversarial Networks (2019, background): “§ (Arjovsky et al., 2017), or adding regularization terms (Gulrajani et al., 2017; Mescheder et al., 2017, 2018).”
StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets (2022, methodology): “Other than spectral normalization, Projected GANs do not use additional regularization such as gradient penalties [Mescheder et al. 2018].”
A Variational Inequality Perspective on Generative Adversarial Networks (2018, methodology): “Regarding the averaging technique, some recent work appear to have already successfully used geometric averaging (7) for GANs in practice, but only brieﬂy mention it (Karras et al., 2018; Mescheder et al., 2018).”
Catastrophic forgetting and mode collapse in GANs (2020, methodology): “This can also be seen in GAN-R1 trained on the 8 Gaussian dataset (Fig.”