Improved Training of Wasserstein GANs
By Ishaan Gulrajani, Faruk Ahmed, Martín Arjovsky, Vincent Dumoulin, Aaron C. Courville. 2017
11,031 citations recorded by the source corpus.
Generative Adversarial Networks (GANs) are powerful generative models, but suffer from training instability. The recently proposed Wasserstein GAN (WGAN) makes progress toward stable training of GANs, but sometimes can still generate only low-quality samples or fail to converge. We find that these problems are often due to the use of weight clipping in WGAN to enforce a Lipschitz constraint on the critic, which can lead to undesired behavior. We propose an alternative to clipping weights: penalize the norm of gradient of the critic with respect to its input. Our proposed method performs better than standard WGAN and enables stable training of a wide variety of GAN architectures with almost no hyperparameter tuning, including 101-layer ResNets and language models over discrete data. We also achieve high quality generations on CIFAR-10 and LSUN bedrooms.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models, Optimization & Training
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Sergey Levine
- Mohit Bansal
- Quoc V. Le
- Jianfeng Gao
- Percy Liang
- Ruslan Salakhutdinov
- Dacheng Tao
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Ming-Hsuan Yang
- Trevor Darrell
- Xiaogang Wang
- Jitendra Malik
- Bernt Schiele
- Andrea Vedaldi
In Generative Models
- Stefano Ermon
- Jun-Yan Zhu
- Richard Zhang
- Arash Vahdat
- Yoshua Bengio
- Eli Shechtman
- Jiaming Song
- Karsten Kreis
In Optimization & Training
- Yoshua Bengio
- Simon Lacoste-Julien
- Quoc V. Le
- Sergey Levine
- Stefano Ermon
- Frank Hutter
- Gauthier Gidel
- Pieter Abbeel
Citations in the corpus
- MaskGAN: Better Text Generation via Filling in the ______ (2018, methodology): “WGAN-GP (Gulrajani et al., 2017) avoids the issue of dealing with backpropagating through discrete nodes by generating text in a one-shot manner using a 1D convolutional network.”
- Are GANs Created Equal? A Large-Scale Study (2017, background): “The gradient norm is evaluated on points obtained by linear interpolation between data points and generated samples where the optimal discriminator should have unit gradient norm [10].”
- Evaluating Lossy Compression Rates of Deep Generative Models (2020, methodology): “We trained shallow and deep GANs with Gradient Penalty (GAN-GP) (Gulrajani et al., 2017) with the code size d ∈ { 2 , 5 , 10 , 100 } on MNIST.”
- Improved Techniques for Training Score-Based Generative Models (2020, background): “6 WGAN-GP [19] 3.”
- Training Generative Adversarial Networks with Limited Data (2020, methodology): “We found WGAN-GP to be quite unstable in our baseline configuration, which necessitated us to disable mixed-precision training and lazy regularization, as well as to settle for a considerably lower learning rate η = 0 .”
- Your GAN is Secretly an Energy-based Model and You Should use Discriminator Driven Latent Sampling (2020, background): “Previous work on GANs focused on improving stability and mode dropping issues (Metz et al., 2016; Che et al., 2016; Arjovsky et al., 2017; Gulrajani et al., 2017; Roth et al., 2017), which are believed to be the main difficulties in training GANs.”
- Analyzing and Improving the Image Quality of StyleGAN (2019, background): “Skip connections [29, 19], residual networks [15, 14, 26], and hierarchical methods [6, 41, 42] have proven highly successful also in the context of generative methods.”
- Flow Contrastive Estimation of Energy-Based Models (2019, neutral): “56 WGAN-GP [20] 40.”
- Generative Modeling by Estimating Gradients of the Data Distribution (2019, neutral): “58 WGAN-GP [18] 7.”
- A Style-Based Generator Architecture for Generative Adversarial Networks (2018, methodology): “For the FFHQ dataset, we switch from WGAN-GP to the non-saturating loss [22] with R 1 regularization [44] using γ = 10 .”
- Large Scale GAN Training for High Fidelity Natural Image Synthesis (2018, methodology): “This leads to two conclusions: first, as has been noted in previous works (Miyato et al., 2018; Gulrajani et al., 2017; Zhang et al., 2018), D must remain optimal with respect to G both for stability and to provide useful gradient information.”
- Spectral Normalization for Generative Adversarial Networks (2018, methodology): “…by augmenting the adversarial loss function with the following regularizer function: where λ > 0 is a balancing coefficient and ˆ x is: Using this augmented objective function, Gulrajani et al. (2017) succeeded in training a GAN based on ResNet (He et al., 2016) with an impressive performance.”
- Progressive Growing of GANs for Improved Quality, Stability, and Variation (2017, methodology): “We will do this by building on top of a previous state-of-theart loss function (WGAN-GP) and training configuration (Gulrajani et al., 2017) in an unsupervised setting using CELEBA (Liu et al.”
- A-NICE-MC: Adversarial Training for MCMC (2017, methodology): “All the models are trained with the gradient penalty objective for Wasserstein GANs [17, 15], where = 1/3, B = 4 and M = 3.”
- Generative Adversarial Networks (2021, neutral): “In previous works that stabilized GAN training ([21, 4, 56]) it was emphasized that D should be a K-Lipshitz continuous function, forcing it not to change rapidly.”
- Image Super-Resolution via Iterative Refinement (2021, background): “Nevertheless, the inner-outer loop optimization often requires tricks to stabilize training [2, 15], and conditional tasks like super-resolution usually require an auxiliary consistencybased loss to avoid mode collapse [25].”
- This Face Does Not Exist... But It Might Be Yours! Identity Leakage in Generative Models (2020, background): “The results and future promise of these generative models have led to heavy academic intrigue [22, 11].”