Many Paths to Equilibrium: GANs Do Not Need to Decrease a Divergence At Every Step

By William Fedus, Mihaela Rosca, Balaji Lakshminarayanan, Andrew M. Dai, Shakir Mohamed, Ian J. Goodfellow. 2017

218 citations recorded by the source corpus.

Generative adversarial networks (GANs) are a family of generative models that do not minimize a single training criterion. Unlike other generative models, the data distribution is learned via a game between a generator (the generative model) and a discriminator (a teacher providing training signal) that each minimize their own cost. GANs are designed to reach a Nash equilibrium at which each player cannot reduce their cost without changing the other players' parameters. One useful approach for the theory of GANs is to show that a divergence between the training distribution and the model distribution obtains its minimum value at equilibrium. Several recent research directions have been motivated by the idea that this divergence is the primary guide for the learning process and that every step of learning should decrease the divergence. We show that this view is overly restrictive. During GAN training, the discriminator provides learning signal in situations where the gradients of the divergences between distributions would not be useful. We provide empirical counterexamples to the view of GAN training as divergence minimization. Specifically, we demonstrate that GANs are able to learn distributions in situations where the divergence minimization point of view predicts they would fail. We also show that gradient penalties motivated from the divergence minimization perspective are equally helpful when applied in other contexts in which the divergence minimization perspective does not predict they would be helpful. This contributes to a growing body of evidence that GAN training may be more usefully viewed as approaching Nash equilibria via trajectories that do not necessarily minimize a specific divergence at each step.

Subfields: Economics and Econometrics, Renewable Energy, Sustainability and the Environment, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Economics and Econometrics

In Renewable Energy, Sustainability and the Environment

In Generative Models

Citations in the corpus

Are GANs Created Equal? A Large-Scale Study (2017, neutral): “Note that the gradient norm can also be evaluated between fake and real points, similarly to WGAN GP, and added to either MM GAN or NS GAN [7].”
Large Scale GAN Training for High Fidelity Natural Image Synthesis (2018, methodology): “Much recent research has accordingly focused on modiﬁcations to the vanilla GAN procedure to impart stability, drawing on a growing body of empirical and theoretical insights (Nowozin et al., 2016; Sønderby et al., 2017; Fedus et al., 2018).”
Diagnosing and Enhancing VAE Models (2019, background): “, 2017), NS GAN (Fedus et al., 2017), DRAGAN (Kodali et al.”
Demystifying MMD GANs (2018, methodology): “…although Goodfellow et al. (2014) motivated GANs as estimating the Jensen-Shannon divergence, they can also be viewed as minimizing the IPM deﬁned by the classiﬁer family (Arora et al., 2017; Liu et al., 2017), thus motivating applying the gradient penalty to original GANs (Fedus et al., 2018).”
A Variational Inequality Perspective on Generative Adversarial Networks (2018, background): “This is the case of the non-saturating formulation of GANs [Goodfellow et al., 2014, Fedus et al., 2018], the DCGANs [Radford et al., 2016].”
Recent Progress on Generative Adversarial Networks (GANs): A Survey (2019, background): “Odena et al. [15] and Fedus et al. [38] used this evaluation metric to determine the diversity of generated data.”
Catastrophic forgetting and mode collapse in GANs (2020, background): “[19] observed that on many real world datasets, the distance between pg and pr (measured in KL-divergence and Jensen-Shannon divergence) does not decrease as t increases.”
The GAN Landscape: Losses, Architectures, Regularization, and Normalization (2018, methodology): “A study of the effects of gradient-norm regularization in GANs was recently presented in Fedus et al. (2018).”
Distribution Matching in Variational Inference (2018, methodology): “In practice, the model and the estimator are trained jointly using alternating gradient descent (Fedus et al., 2018, Goodfellow et al., 2014).”
The relativistic discriminator: a key element missing from standard GAN (2018, background): “The constraint of WGAN-GP (i.e., discriminator with gradient norm equal to 1 around real and fake data) has been shown to be beneﬁcial in SGAN [Fedus et al., 2017] (along with a very similar gradient penalty by Kodali et al. [2017]).”
Feature Quantization Improves GAN Training (2020, background): “Much recent research has accordingly focused on improving its stability, drawing on a growing body of empirical and theoretical insights (Nowozin et al., 2016; Li et al., 2017a; Zhu et al., 2017; Fedus et al., 2017).”
Adversarial Transfer Learning (2018, background): “[34] found adding a gradient penalty benefits the original NS-GAN as well.”
A Variational Inequality Perspective on Generative Adversarial Nets (2018, background): “This is the case of the non-saturating formulation of GANs (Goodfellow et al., 2014; Fedus et al., 2017), the DCGANs (Radford et al., 2016), the gradient penalty5 for WGANs (Gulrajani et al., 2017).”
Diffusion-GAN: Training GANs with Diffusion (2022, methodology): “Fedus et al. (2018) provide a rich set of empirical evidence to discourage viewing GANs purely from the perspective of minimizing a specific divergence at each training step and emphasize the important role played by gradient penalties on stabilizing GAN training.”
A Large-Scale Study on Regularization and Normalization in GANs (2018, background): “We summarize the fixed hyperparameter settings in Table 1 which contains the “good” parameters reported in recent publications (Fedus et al., 2018; Miyato et al., 2018; Gulrajani et al., 2017).”
Discriminator optimal transport (2019, background): “(5) Theoretically speaking, it seems better to take g(u) = f(u), which is called minimax GAN [22] to guarantee pG = p at the equilibrium as proved in [1].”
A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications (2020, methodology): “References [30], [223] used MS-SSIM to measure the diversity of fake data.”