Towards Principled Methods for Training Generative Adversarial Networks

By Martín Arjovsky, L. Eon Bottou. 2017

2,278 citations recorded by the source corpus.

The goal of this paper is not to introduce a single algorithm or method, but to make theoretical steps towards fully understanding the training dynamics of generative adversarial networks. In order to substantiate our theoretical analysis, we perform targeted experiments to verify our assumptions, illustrate our claims, and quantify the phenomena. This paper is divided into three sections. The first section introduces the problem at hand. The second section is dedicated to studying and proving rigorously the problems including instability and saturation that arize when training generative adversarial networks. The third section examines a practical and theoretically grounded direction towards solving these problems, while introducing new tools to study them.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Generative Models

Citations in the corpus

Training Generative Adversarial Networks with Limited Data (2020, background): “The key problem with small datasets is that the discriminator overfits to the training examples; its feedback to the generator becomes meaningless and training starts to diverge [2, 48].”
Spectral Normalization for Generative Adversarial Networks (2018, methodology): “A particularly successful works in this array are (Qi, 2017; Arjovsky et al., 2017; Gulrajani et al., 2017), which proposed methods to control the Lipschitz constant of the discriminator by adding regularization terms deﬁned on input examples x .”
Progressive Growing of GANs for Improved Quality, Stability, and Variation (2017, background): “When we measure the distance between the training distribution and the generated distribution, the gradients can point to more or less random directions if the distributions do not have substantial overlap, i.e., are too easy to tell apart (Arjovsky & Bottou, 2017).”
Projected GANs Converge Faster (2021, background): “Indeed, we confirm that a naïve application of this idea does not lead to state-of-the-art results (Section 4) as strong pretrained features enable the discriminator to dominate the two-player game, resulting in vanishing gradients for the generator [2].”
Improved Training of Wasserstein GANs (2017, background): “It is satisfied in the case that Pr and Pg have supports that intersect in a set of measure 0, such as when they are supported by two low dimensional manifolds that don’t perfectly align [1].”
Learning deep representations by mutual information estimation and maximization (2018, background): “As discussed in Arjovsky & Bottou (2017), if P and Q θ have support on a low-dimensional manifolds on X , unless they are perfectly aligned, there exists a discriminator that will be able to perfectly distinguish between samples coming from P and Q θ , which means that U ψ, P and U ψ, Q must also…”
Many Paths to Equilibrium: GANs Do Not Need to Decrease a Divergence At Every Step (2017, methodology): “Roth et al. [21] proposed a gradient penalty aiming to smooth the discriminator of f -GANs (including the minimax GAN), which we refer to as f -GAN-GP, inspired by Sønderby et al. [23] and Arjovsky and Bottou [1].”
Coulomb GANs: Provably Optimal Nash Equilibria via Potential Fields (2017, background): “The restriction to local environments hampers GAN learning signiﬁcantly [1, 2].”
Gradient descent GAN optimization is locally stable (2017, background): “Arjovsky and Bottou [2017] provide important insights into instability that arises when the supports of the generated distribution and the true distribution are disjoint. In contrast, in this paper we delve into an equally important question of whether the updates are stable even when the generator ”
The Numerics of GANs (2017, background): “When we are trying to minimize an f-divergence and the dimensionality of the generator distribution is misspecified, this might not be the case [3].”
Comparison of Maximum Likelihood and GAN-based training of Real NVPs (2017, methodology): “Wasserstein GAN (WGAN) (Arjovsky et al., 2017) uses a critic instead of a discriminator.”
Wasserstein GAN (2017, background): “In this paper, we direct our attention on the various ways to measure how close the model distribution and the real distribution are, or equivalently, on the various ways to deﬁne a distance or divergence ρ ( P θ , P r ).”
StackGAN: Text to Photo-Realistic Image Synthesis with Stacked Generative Adversarial Networks (2016, background): “Several techniques [23, 29, 18, 1, 3] have been proposed to stabilize the training process and generate compelling results.”
Which Training Methods for GANs do actually Converge? (2018, methodology): “As a result, a lot of recent research has focused on finding better training algorithms (Arjovsky et al., 2017; Gulrajani et al., 2017; Kodali et al., 2017; Sønderby et al., 2016; Roth et al., 2017) for GANs as well as gaining better theoretically understanding of their training dynamics (Arjovsky e”
Demystifying MMD GANs (2018, methodology): “Recent notable work has made use of the IPM representation of the MMD to employ the same witness function regularization strategies as Arjovsky et al. (2017) and Gulrajani et al. (2017), effectively corresponding to an additional constraint on the MMD function class.”
Amortised MAP Inference for Image Super-resolution (2016, neutral): “We also refer to parallel work by Arjovsky & Bottou (2017) proposing a similar method.”
Geometric GAN (2017, background): “In particular, the authors in [6] have iden-tiﬁed the following sources of the difﬁculties: 1) when the discriminator becomes accurate, the a r X i v : 1705 .”
Photographic Image Synthesis with Cascaded Refinement Networks (2017, background): “Nevertheless, as observed in recent work and widely known in the folklore, GANs “remain remarkably difficult to train” and “approaches to attacking this problem still rely on heuristics that are extremely sensitive to modifications” [1].”