Stabilizing Training of Generative Adversarial Networks through Regularization

By Kevin Roth, Aurélien Lucchi, Sebastian Nowozin, Thomas Hofmann. 2017

481 citations recorded by the source corpus.

Deep generative models based on Generative Adversarial Networks (GANs) have demonstrated impressive sample quality but in order to work they require a careful choice of architecture, parameter initialization, and selection of hyper-parameters. This fragility is in part due to a dimensional mismatch or non-overlapping support between the model distribution and the data distribution, causing their density ratio and the associated f -divergence to be undefined. We overcome this fundamental limitation and propose a new regularization approach with low computational cost that yields a stable GAN training procedure. We demonstrate the effectiveness of this regularizer accross several architectures trained on common benchmark image generation tasks. Our regularization turns GAN models into reliable building blocks for deep learning.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Generative Models, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Generative Models

In Optimization & Training

Citations in the corpus

Your GAN is Secretly an Energy-based Model and You Should use Discriminator Driven Latent Sampling (2020, background): “Previous work on GANs focused on improving stability and mode dropping issues (Metz et al., 2016; Che et al., 2016; Arjovsky et al., 2017; Gulrajani et al., 2017; Roth et al., 2017), which are believed to be the main difficulties in training GANs.”
Learning Non-Convergent Non-Persistent Short-Run MCMC Toward Energy-Based Model (2019, neutral): “, injecting additive noises εi ∼ N(0,σ(2)I) to observed examples xi← xi + εi [42, 39].”
A U-Net Based Discriminator for Generative Adversarial Networks (2020, background): “Note that we do not modify the generator in any way, and our work is orthogonal to the ongoing research on architectural changes of the generator [20, 27], divergence measures [25, 1, 37], and regularizations [40, 15, 34].”
Many Paths to Equilibrium: GANs Do Not Need to Decrease a Divergence At Every Step (2017, methodology): “[21] proposed a gradient penalty aiming to smooth the discriminator of f -GANs (including the minimax GAN), which we refer to as f -GAN-GP, inspired by Sønderby et al.”
Gradient descent GAN optimization is locally stable (2017, methodology): “Of particular similarity to the methodology we propose here are the works by Roth et al. [2017] and Mescheder et al. [2017].”
Which Training Methods for GANs do actually Converge? (2018, methodology): “As a result, a lot of recent research has focused on finding better training algorithms (Arjovsky et al., 2017; Gulrajani et al., 2017; Kodali et al., 2017; Sønderby et al., 2016; Roth et al., 2017) for GANs as well as gaining better theoretically understanding of their training dynamics (Arjovsky e”
Unsupervised Model Adaptation for Continual Semantic Segmentation (2020, background): “Adversarial learning requires delicate optimization initialization, architecture engineering, and careful selection of hyper-parameters to be stable (Roth et al. 2017).”
Complementary Learning for Overcoming Catastrophic Forgetting Using Experience Replay (2019, background): “However, adversarial learning is known to require deliberate architecture design and selection of hyper-parameters [Roth et al., 2017], and can suffer from mode collapse [Srivastava et al., 2017].”
Gradient Surgery for Multi-Task Learning (2020, background): “Further, the general idea of gradient surgery may be an important ingredient for alleviating a broader class of optimization challenges in deep learning, such as the challenges in the stability challenges in two-player games (Roth et al., 2017) and multi-agent optimizations (Nedic & Ozdaglar, 2009).”
The GAN Landscape: Losses, Architectures, Regularization, and Normalization (2018, background): “Furthermore, the penalty can be scaled by the "confidence" of the discriminator in the context of f-divergences [24].”
MoCoGAN: Decomposing Motion and Content for Video Generation (2017, background): “[30] proposed a special gradient penalty to further stabilize training.”
Boundary-Seeking Generative Adversarial Networks (2017, methodology): “All results use gradient norm regularization (Roth et al., 2017) to ensure stability.”
Differentiable Augmentation for Data-Efficient GAN Training (2020, methodology): “To stabilize the training dynamics, researchers have proposed several techniques including the instance noise [37], Jensen-Shannon regularization [32], gradient penalties [12, 25], spectral normalization [26], adversarial defense regularization [48], and consistency regularization [46].”
Adversarial Latent Autoencoders (2020, methodology): “We utilize R 1 [44, while not converged do 3: Step I. Update E , and D 4: x ← Random mini-batch from dataset 5: z ← Samples from prior N (0 ,I ) 6: Step II.”
Consistency Regularization for Generative Adversarial Networks (2019, methodology): “7 times faster than gradient based regularization techniques, including DR, GP and JSR, which need to compute the gradient of the gradient norm (cid:107)∇ x ( D ) (cid:107) .”
PacGAN: The Power of Two Samples in Generative Adversarial Networks (2017, background): “Recently, other innovative choices for regularizations have been proposed, including weight clipping [1], gradient regularization [20, 45, 50], Tikhonov regularizer for training-with-noise [58], adding noise to the gradient [21], and spectral-norm regularization [49].”
An Online Learning Approach to Generative Adversarial Networks (2017, methodology): “Note that the DCGAN architecture, when trained using standard techniques, still suffers from instabilities and mode collapse (Nagarajan & Kolter, 2017; Roth et al., 2017).”
Quantum Generative Adversarial Networks for learning and loading random distributions (2019, methodology): “[40] K.”