Consistency Regularization for Generative Adversarial Networks

By Han Zhang, Zizhao Zhang, Augustus Odena, Honglak Lee. 2019

311 citations recorded by the source corpus.

Generative Adversarial Networks (GANs) are known to be difficult to train, despite considerable research effort. Several regularization techniques for stabilizing training have been proposed, but they introduce non-trivial computational overheads and interact poorly with existing techniques like spectral normalization. In this work, we propose a simple, effective training stabilizer based on the notion of consistency regularization---a popular technique in the semi-supervised learning literature. In particular, we augment data passing into the GAN discriminator and penalize the sensitivity of the discriminator to these augmentations. We conduct a series of experiments to demonstrate that consistency regularization works effectively with spectral normalization and various GAN architectures, loss functions and optimizer settings. Our method achieves the best FID scores for unconditional image generation compared to other regularization methods on CIFAR-10 and CelebA. Moreover, Our consistency regularized GAN (CR-GAN) improves state-of-the-art FID scores for conditional generation from 14.73 to 11.48 on CIFAR-10 and from 8.73 to 6.66 on ImageNet-2012.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Training Generative Adversarial Networks with Limited Data (2020, background): “In contrast, a GAN trained under similar dataset augmentations learns to generate the augmented distribution [50, 53].”
A U-Net Based Discriminator for Generative Adversarial Networks (2020, methodology): “The progress is attributed to large-scale training [32, 5], architectural modifications [50, 19, 20, 27], and improved training stability via the use of different regularization techniques [34, 51].”
Cross-Modal Contrastive Learning for Text-to-Image Generation (2021, methodology): “The hinge loss has been used in state-of-the-art GANs for image generation [32, 60, 4, 63].”
vCLIMB: A Novel Video Class Incremental Learning Benchmark (2022, neutral): “For example, such methods have been shown to improve the generation of images from a few samples [43, 45].”
Image Augmentations for GAN Training (2020, methodology): “For example, the baseline SNDCGAN trained without any image augmentation achieves 24.73 in FID [42], while translation , even with its smallest strength, gets 31.03.”
Freeze the Discriminator: a Simple Baseline for Fine-Tuning GANs (2020, methodology): “…[45, 32, 44], learning meta-knowledge for quick adaptation to a target domain [24, 47, 42], using an auxiliary task to facilitate training [7, 26, 48, 49], improving an inference procedure of suboptimal models [2, 39, 29, 38], using an expressive prior distribution [13], actively choosing…”
TransGAN: Two Transformers Can Make One Strong GAN (2021, methodology): “Our ambitious goal is clearly distinguished from the previous works that only applied self-attention or transformer encoder block in conjunction with CNN-based generative models (Zhang et al., 2019a; Esser et al., 2020).”
Domain Adaptive Ensemble Learning (2020, methodology): “., virtual adversarial training (VAT) [38], mean teacher [50], MixMatch [6] and its recent follow-ups [5,49]. Other than SSL, CR has also been applied to applications such as stabilizing GAN training [62]. As an unsupervised objective, it is natural to use CR for dealing with unlabeled data in UDA. ”
Robust Learning by Self-Transition for Handling Noisy Labels (2020, background): “Furthermore, to avoid the potential risk of overfitting to a small initial seed, which is typically observed with a very high noise rate, MORPH adds consistency regularization J (Θt ) [16, 45] to the supervised loss in Eq.”
Regularizing Generative Adversarial Networks under Limited Data (2021, methodology): “To address these issues, recent meth-ods [30, 68, 80, 82, 83, 84] exploit data augmentation as a mean to increase data diversity, hence preventing the GAN models from overﬁtting the training data.”
Data-Efficient Instance Generation from Instance Discrimination (2021, background): “Data Augmentation in GANs.”
An Overview of Deep Semi-Supervised Learning (2020, background): “Which are subsequently improved in several ways [174, 105, 175], such as using spectral normalization [105] on both the generator and the discriminator, or consistency regularization on the discriminator [175].”
Tackling the Generative Learning Trilemma with Denoising Diffusion GANs (2021, background): “…adversarial loss, our work is related to recent advances in improving the sample quality and diversity of GANs, including data augmentation (Zhao et al., 2020; Karras et al., 2020a), consistency regularization (Zhang et al., 2020; Zhao et al., 2021) and entropy regularization (Dieng et al., 2019).”
Partition-Guided GANs (2021, background): “Others [9, 49, 84] incorporated regularization on top of the augmentations, for example CRGAN [83] enforces consistency for different image perturbations.”