Training Generative Adversarial Networks with Limited Data

By Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, Timo Aila. 2020

2,283 citations recorded by the source corpus.

Training generative adversarial networks (GAN) using too little data typically leads to discriminator overfitting, causing training to diverge. We propose an adaptive discriminator augmentation mechanism that significantly stabilizes training in limited data regimes. The approach does not require changes to loss functions or network architectures, and is applicable both when training from scratch and when fine-tuning an existing GAN on another dataset. We demonstrate, on several datasets, that good results are now possible using only a few thousand training images, often matching StyleGAN2 results with an order of magnitude fewer images. We expect this to open up new application domains for GANs. We also find that the widely used CIFAR-10 is, in fact, a limited data benchmark, and improve the record FID from 5.59 to 2.42.

Subfields: Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Generative Models

Citations in the corpus

Generative Adversarial Networks (2021, positive): “Data augmentation can be also helpful in the training of the GANs themselves [31, 78], which allows to train them with only few examples and still get high quality generated data (e.”
Projected GANs Converge Faster (2021, methodology): “As in [25], we measure training progress based on the number of real images shown to the discriminator (Imgs).”
Score-based Generative Modeling in Latent Space (2021, background): “[84] Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila.”
On Buggy Resizing Libraries and Surprising Subtleties in FID Calculation (2021, background): “Here, we evaluate a pretrained StyleGAN2 generator [38] trained on FFHQ (1024), MetFaces (1024), and AFHQ (512) dataset images, and calculate FID with 50,000 images.”
Taming Transformers for High-Resolution Image Synthesis (2020, neutral): “CelebAHQ results reproduced from [1, 47, 67, 22], FFHQ from [58, 28].”
StyleGAN-NADA (2021, methodology): “Such works typically employ data aug-mentations [20,50,64,65] or empower the discriminator to better learn from existing data using auxiliary tasks [29,60].”
StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery (2021, methodology): “Here we use StyleGAN-ada [5] pretrained on AFHQ wild [2], which contains wolves, lions, tigers and foxes.”
Elucidating the Design Space of Diffusion-Based Generative Models (2022, methodology): “To prevent potential overfitting that often plagues diffusion models with smaller datasets, we borrow an augmentation pipeline from the GAN literature [25].”
StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets (2022, methodology): “We follow [Karras et al. 2020b] and use the original latent space W .”
Differentiable Augmentation for Data-Efficient GAN Training (2020, methodology): “Concurrent with our work, several methods [16, 41, 51] independently proposed data augmentation for training GANs.”
Alias-Free Generative Adversarial Networks (2021, methodology): “M ET F ACES was trained using adaptive discriminator augmentation (ADA) [32], starting from the corresponding FFHQ snapshot with the lowest FID.”
BigVGAN: A Universal Neural Vocoder with Large-Scale Training (2022, methodology): “Data augmentation Data augmentation is one of major methods that improve model generalization, which is also repeatedly found to be valuable in GAN literature (Karras et al., 2020).”
Learning Energy-Based Models by Diffusion Recovery Likelihood (2020, neutral): “4 7.86 ± .07 SNGAN (Miyato et al., 2018) 21.7 8.22 ± .05 SNGAN-DDLS (Che et al., 2020) 15.42 9.09 ± .10 StyleGAN2-ADA (Karras et al., 2020) 3.26 9.74 ± .05”
Designing an encoder for StyleGAN image manipulation (2021, background): “Notably, it has been shown that StyleGAN [19, 20, 21], with its novel style-based architecture, contains a semantically rich latent space that can be used to perform diverse image manipulations.”
Encoding in Style: a StyleGAN Encoder for Image-to-Image Translation (2020, methodology): “While our method does require a pretrained StyleGAN generator, recent works [19, 34] have shown that such a generator can be easily trained with signiﬁcantly fewer examples than required in the past.”