On the regularization of Wasserstein GANs

By Henning Petzka, Asja Fischer, Denis Lukovnikov. 2017

221 citations recorded by the source corpus.

Since their invention, generative adversarial networks (GANs) have become a popular approach for learning to model a distribution of real (unlabeled) data. Convergence problems during training are overcome by Wasserstein GANs which minimize the distance between the model and the empirical distribution in terms of a different metric, but thereby introduce a Lipschitz constraint into the optimization problem. A simple way to enforce the Lipschitz constraint on the class of functions, which can be modeled by the neural network, is weight clipping. It was proposed that training can be improved by instead augmenting the loss by a regularization term that penalizes the deviation of the gradient of the critic (as a function of the network's input) from one. We present theoretical arguments why using a weaker regularization term enforcing the Lipschitz constraint is preferable. These arguments are supported by experimental results on toy data sets.

Subfields: Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

Citations in the corpus

Recent Progress on Generative Adversarial Networks (GANs): A Survey (2019, background): “In Sections 3 and 4, [27]–[29], [31]–[33] put forward their own solutions for more stable training.”
Simple and Scalable Epistemic Uncertainty Estimation Using a Single Deep Deterministic Neural Network (2020, neutral): “Follow up work has analysed the penalty in more detail and concluded that, contrary to our case, for training Wasserstein GANS the one-sided penalty is preferable theoretically and practically (Jolicoeur-Martineau & Mitliagkas, 2019; Petzka et al., 2017).”
Uncertainty Estimation Using a Single Deep Deterministic Neural Network (2020, result): “Follow up work has analysed the penalty in more detail and concluded that, contrary to our case, for training Wasserstein GANS the one-sided penalty is preferable theoretically and practically (Jolicoeur-Martineau & Mitliagkas, 2019; Petzka et al., 2017).”
Improving GAN with neighbors embedding and gradient matching (2018, unknown): “…are weight-clipping (Ar-jovsky and Bottou 2017), gradient penalty constraints (Gul-rajani et al. 2017; Roth et al. 2017; Kodali et al. 2017; Petzka, Fischer, and Lukovnicov 2017; Liu 2018), consensus constraint, (Mescheder, Nowozin, and Geiger 2017; Mescheder, Geiger, and Nowozin 2018), or…”
PassGAN: A Deep Learning Approach for Password Guessing (2017, unknown): “[25], there have been several improvements on GANs [3, 4, 6–8, 10, 14, 16, 27, 35, 36, 42, 47, 53, 55, 56, 58–60, 63, 66, 69, 74, 76, 80, 87, 90, 94, 96], where each new paper provides novel improvements in the domain.”
An Improved Self-supervised GAN via Adversarial Training (2019, methodology): “Therefore, many regularizations have been proposed to reduce this problem, and the most popular technique is to enforce (or towards) Lipschitz condition of the discriminator by weight-clipping [1], gradient penalty constraints [14, 29, 17, 27, 19], consensus constraint, [22, 21], or spectral norm [2”
Sphere Generative Adversarial Network Based on Geometric Moment Matching (2019, methodology): “In conventional GANs based on the Wesserstein distance [2, 9, 24, 37], objective functions are designed as a dual form by the Kantorovich-Rubinstein duality theorem.”
Text to Image Synthesis Using Generative Adversarial Networks (2018, methodology): “Moreover, empirical and theoretical evidence [26] shows that, under this softer regularisation term, convergence is faster and more stable.”
On Data Augmentation for GAN Training (2020, background): “penalty constraints [21], [22], [23], [24], [25] and spectral norm [26].”
A Survey on Generative Adversarial Networks: Variants, Applications, and Training (2020, methodology): “Despite the early efforts towards that are broadly used at current such as Inception Scores (IS) [196], Mode Score (MS) [225], Kernel Inception Distance (KID) [226], Fréchet Inception Distance (FID) [198], Multi-Scale Structural Similarity (MS-SSIM) [227], Classifier Two-Sample Tests (C2ST) [228], W”
A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications (2020, background): “Other papers related to WGAN can be found in [173]–[178].”
Connections between Support Vector Machines, Wasserstein distance and gradient-penalty GANs (2019, methodology): “However, this is not equivalent to estimating Wasserstein’s distance since we are not sampling from π∗ and f∗ does not need to be differentiable everywhere (Petzka et al., 2017).”
Self-supervised GAN: Analysis and Improvement with Multi-class Minimax Game (2019, methodology): “These methods include weight-clipping [1], gradient penalty constraints [13, 39, 21, 36, 27] and spectral norm [31].”
Discriminator optimal transport (2019, background): “Since from the first successful implementation by convolutional neural nets [2], most literatures concentrate on how to improve the unstable optimization procedures including changing objective functions [3, 4, 5, 6, 7, 8], adding penalty terms [9, 10, 11], techniques on optimization precesses thems”
Leveraging Frequency Analysis for Deep Fake Image Recognition (2020, methodology): “, 2016), utilizing the Wasserstein distance (Arjovsky et al., 2017; Gulrajani et al., 2017; Petzka et al., 2018), spectral normalization (Miyato et al.”
Generative Adversarial Networks (GANs) (2020, background): “On the other hand, [101] have present a theoretical claim that BWGAN is not good for GANs training and then propose a less restrictive regularization.”
Exactly Computing the Local Lipschitz Constant of ReLU Networks (2020, methodology): “For the Wasserstein GAN formulation (Arjovsky et al., 2017) the discriminator is required to have a bounded Lipschitz constant, and there are several techniques to enforce this (Arjovsky et al., 2017; Cisse et al., 2017; Petzka et al., 2017).”