Improving the Improved Training of Wasserstein GANs: A Consistency Term and Its Dual Effect

By Xiang Wei, Boqing Gong, Zixia Liu, Wei Dang Lu, Liqiang Wang. 2018

281 citations recorded by the source corpus.

Despite being impactful on a variety of problems and applications, the generative adversarial nets (GANs) are remarkably difficult to train. This issue is formally analyzed by \cite{arjovsky2017towards}, who also propose an alternative direction to avoid the caveats in the minmax two-player training of GANs. The corresponding algorithm, called Wasserstein GAN (WGAN), hinges on the 1-Lipschitz continuity of the discriminator. In this paper, we propose a novel approach to enforcing the Lipschitz continuity in the training procedure of WGANs. Our approach seamlessly connects WGAN with one of the recent semi-supervised learning methods. As a result, it gives rise to not only better photo-realistic samples than the previous methods but also state-of-the-art semi-supervised learning results. In particular, our approach gives rise to the inception score of more than 5.0 with only 1,000 CIFAR-10 images and is the first that exceeds the accuracy of 90% on the CIFAR-10 dataset using only 4,000 labeled images, to the best of our knowledge.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Semi-Supervised Sequence Modeling with Cross-View Training (2018, background): “…Miyato et al., 2017b) rely on adding continuous noise and applying image-speciﬁc transformations like cropping to inputs, GANs (Salimans et al., 2016; Wei et al., 2018) are very difﬁcult to train on text due to its discrete nature, and mixup (Zhang et al., 2018; Verma et al., 2018) requires a way…”
Consistency Regularization for Generative Adversarial Networks (2019, neutral): “Previous work (Arjovsky & Bottou, 2017; Miyato et al., 2018a; Odena et al., 2017; Chen et al., 2019; Wei et al., 2018) has shown that interventions focused on the discriminator can mitigate stability issues.”
A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music (2018, background): “12 and 13 in Appendix E for a baseline comparison. millions of pixels (Karras et al., 2017), generating synthetic audio with hundreds of thousands of timesteps (van den Oord et al., 2016a), and achieving state-of-the-art performance on semi-supervised learning tasks (Wei et al., 2018).”
MedGAN: Medical Image Translation using GANs (2018, background): “researchers have proposed a series of modifications to the adversarial loss through the Wasserstein distance [27]–[29] in W-GAN and the MMD distance [30], [31] in MMDGAN.”
Training GANs with Stronger Augmentations via Contrastive Discriminator (2021, background): “Stabilizing the GAN dynamics has been extensively studied in the literature (Arjovsky et al., 2017; Gulrajani et al., 2017; Miyato et al., 2018; Wei et al., 2018; Jolicoeur-Martineau, 2019; Chen et al., 2019; Schonfeld et al., 2020), and the idea of incorporating data augmentation techniques has…”
ODE Analysis of Stochastic Gradient Methods with Optimism and Anchoring for Minimax Problems and GANs (2019, unknown): “There are several independent directions for improving the training of GANs such as designing better architectures, choosing good loss functions, or adding appropriate regularizers [54, 2, 60, 1, 20, 64, 59, 37, 38, 41].”
Consistency Regularization for Variational Auto-Encoders (2021, background): “Indeed (Wei et al., 2018; Zhang et al., 2020) show that applying consistency regularization on the discriminator of a GAN—also a classifier—can substantially improve its performance.”
Generative Adversarial Networks (GANs) (2020, background): “While Reference [115] showed that gradient penalty in WGAN-GP cannot support stable training, as GP often does not check the continuity of region near the real data.”
Sphere Generative Adversarial Network Based on Geometric Moment Matching (2019, methodology): “In conventional GANs based on the Wesserstein distance [2, 9, 24, 37], objective functions are designed as a dual form by the Kantorovich-Rubinstein duality theorem.”
Adversarial Distillation of Bayesian Neural Network Posteriors (2018, methodology): “Extensions include using a gradient penalty (Gul-rajani et al., 2017), and ﬁnite differences of gradients (Wei et al., 2018) to enforce this constraint.”
PassGAN: A Deep Learning Approach for Password Guessing (2017, unknown): “[25], there have been several improvements on GANs [3, 4, 6–8, 10, 14, 16, 27, 35, 36, 42, 47, 53, 55, 56, 58–60, 63, 66, 69, 74, 76, 80, 87, 90, 94, 96], where each new paper provides novel improvements in the domain.”
Learning Neural Random Fields with Inclusive Auxiliary Generators (2018, unknown): “13 CT-GAN [38] 8.”
Improved Consistency Regularization for GANs (2020, methodology): “Generative Adversarial Networks (GANs; Goodfellow et al., 2014) are a powerful class of deep generative models, but are known for training difﬁculties (Salimans et al., 2016).”
A Survey on Deep Semi-Supervised Learning (2021, methodology): “CatGAN [32] 1:39ð 0:28Þ 19:58ð 0:58Þ ImprovedGAN [33] 0:93ð 0:07Þ 18:44ð 4:8Þ 8:11ð 1:3Þ 21:83ð 2:01Þ 18:63ð 2:32Þ GoodBadGAN [34] 0:80ð 0:1Þ 4:25ð 0:03Þ 14:41ð 0:30Þ Localized GAN [35] 5:48ð 0:29Þ 4:73ð 0:16Þ 17:44ð 0:25Þ 14:23ð 0:27Þ CT-GAN [36] 0:89ð 0:13Þ 9:98ð 0:21Þ ALI [37] 7:42ð 0:65Þ 19:98ð ”
Discriminator optimal transport (2019, background): “Since from the first successful implementation by convolutional neural nets [2], most literatures concentrate on how to improve the unstable optimization procedures including changing objective functions [3, 4, 5, 6, 7, 8], adding penalty terms [9, 10, 11], techniques on optimization precesses thems”