Differentiable Augmentation for Data-Efficient GAN Training

By Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, Song Han. 2020

694 citations recorded by the source corpus.

The performance of generative adversarial networks (GANs) heavily deteriorates given a limited amount of training data. This is mainly because the discriminator is memorizing the exact training set. To combat it, we propose Differentiable Augmentation (DiffAugment), a simple method that improves the data efficiency of GANs by imposing various types of differentiable augmentations on both real and fake samples. Previous attempts to directly augment the training data manipulate the distribution of real images, yielding little benefit; DiffAugment enables us to adopt the differentiable augmentation for the generated samples, effectively stabilizes training, and leads to better convergence. Experiments demonstrate consistent gains of our method over a variety of GAN architectures and loss functions for both unconditional and class-conditional generation. With DiffAugment, we achieve a state-of-the-art FID of 6.80 with an IS of 100.8 on ImageNet 128x128. Furthermore, with only 20% training data, we can match the top performance on CIFAR-10 and CIFAR-100. Finally, our method can generate high-fidelity images using only 100 images without pre-training, while being on par with existing transfer learning algorithms. Code is available at this https URL.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Training Generative Adversarial Networks with Limited Data (2020, background): “While two of these papers [54, 51] propose essentially the same augmentation mechanism as we do, they study the absence of leak artifacts only empirically.”
Generative Adversarial Networks (2021, positive): “Data augmentation can be also helpful in the training of the GANs themselves [31, 78], which allows to train them with only few examples and still get high quality generated data (e.”
Projected GANs Converge Faster (2021, methodology): “differentiable data-augmentation [72] and adaptive discriminator augmentation [25].”
Large Scale Image Completion via Co-Modulated Generative Adversarial Networks (2021, background): “…al., 2019; Xiong et al., 2019; Yan et al., 2018; Yang et al., 2017; 2019a; Yu et al., 2018; Yu et al., 2019; Zeng et al., 2019; Lahiri et al., 2020; Zhao et al., 2020a; Li et al., 2020; Zhou et al., 2020), among which (Liu et al., 2018; Yu et al., 2019) introduce partial convolution and gated…”
StyleGAN-NADA (2021, background): “Such works typically employ data augmentations [14, 41, 52, 53] or empower the discriminator to better learn from existing data using auxiliary tasks [23, 49].”
StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets (2022, background): “2021] apply differentiable dataaugmentation [Zhao et al. 2020] before F which improves Projected GAN’s performance independent of the dataset size.”
vCLIMB: A Novel Video Class Incremental Learning Benchmark (2022, neutral): “For example, such methods have been shown to improve the generation of images from a few samples [43, 45].”
Adversarial score matching and improved sampling for image generation (2020, methodology): “Still, GANs often have trouble producing data from all possible modes, which limits the diversity of the generated samples.”
Instance-Conditioned GAN (2021, positive): “For COCO-Stuff and ImageNet-LT, we found that using translations with the DiffAugment framework [59] improves FID scores, as the number of training samples is signiﬁcantly smaller than ImageNet (5% and 10% the size of ImageNet, respectively).”
Mind the Gap: Domain Gap Control for Single Shot Domain Adaptation for Generative Adversarial Networks (2021, background): “Another work, DiffAug (Zhao et al., 2020), applies differentiable transformations to the real and generated images for robust training.”
Lafite2: Few-shot Text-to-Image Generation (2022, background): “In the literature, it is known that training and ﬁne-tuning large unconditional generative models with limited samples is not an easy task [13,22,42,43], not to mention that one needs to additionally consider image-text correspondence besides image quality.”
Data-Efficient Instance Generation from Instance Discrimination (2021, background): “Some recent attempts [24, 39, 49, 51, 44] have been made to apply data augmentation to GAN training.”
ContraGAN: Contrastive Learning for Conditional Image Generation (2020, background): “However, GANs with the projection discriminator have overﬁtting issues, which lead to the collapse of adversarial training [21, 9, 22, 23].”
Tackling the Generative Learning Trilemma with Denoising Diffusion GANs (2021, methodology): “Since our model is trained with adversarial loss, our work is related to recent advances in improving the sample quality and diversity of GANs, including data augmentation (Zhao et al., 2020; Karras et al., 2020a), consistency regularization (Zhang et al.”