Image Generation From Small Datasets via Batch Statistics Adaptation

By Atsuhiro Noguchi, Tatsuya Harada. 2019

211 citations recorded by the source corpus.

Thanks to the recent development of deep generative models, it is becoming possible to generate high-quality images with both fidelity and diversity. However, the training of such generative models requires a large dataset. To reduce the amount of data required, we propose a new method for transferring prior knowledge of the pre-trained generator, which is trained with a large dataset, to a small dataset in a different domain. Using such prior knowledge, the model can generate images leveraging some common sense that cannot be acquired from a small dataset. In this work, we propose a novel method focusing on the parameters for batch statistics, scale and shift, of the hidden layers in the generator. By training only these parameters in a supervised manner, we achieved stable training of the generator, and our method can generate higher quality images compared to previous methods without collapsing, even when the dataset is small (~100). Our results show that the diversity of the filters acquired in the pre-trained generator is important for the performance on the target domain. Our method makes it possible to add a new class or domain to a pre-trained generator without disturbing the performance on the original domain. Code is available at github.com/nogu-atsu/small-dataset-image-generation

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Training Generative Adversarial Networks with Limited Data (2020, background): “Several authors have explored this in the context of GANs [44, 45, 34], and Mo et al.”
StyleGAN-NADA (2021, background): “Others introduce new parameters in order to control channelwise statistics [28], leverage a miner network in order to steer GAN sampling towards suitable regions of the latent space [45], add novel regularization terms to the loss function [22, 42] or mix contrastive learning ideas with patchlevel d”
GAN Memory with No Forgetting (2020, methodology): “FiLM imposes simple element-wise affine transformations to latent features of a neural network, showing remarkable effectiveness in various domains [30, 17, 33, 58].”
When, Why, and Which Pretrained GANs Are Useful? (2022, unknown): “A line of works (Wang et al., 2018b; Noguchi & Harada, 2019; Zhao et al., 2020; Mo et al., 2020; Wang et al., 2020; Li et al., 2020) investigate different methods to transfer GANs to new datasets and report significant advantages compared to training from scratch both in terms of generative quality…”
Few-shot Image Generation via Cross-domain Correspondence (2021, methodology): “…model, pre-trained on a large dataset, and adapt it to a target domain with limited data by either making only small changes to the network parameters to preserve as much information as possible [34, 23, 32, 20, 16], or by synthetically increasing the training data via data augmentation [39, 11].”
Regularizing Generative Adversarial Networks under Limited Data (2021, methodology): “Methods Pre-training? 100-shot [16] AnimalFace [13] Obama Grumby cat Panda Cat Dog Scale/shift [12] X 50.”
SVDiff: Compact Parameter Space for Diffusion Fine-Tuning (2023, background): “This approach is inspired by prior work in GAN adaptation showing that constraining the space of trainable parameters can lead to improved performance on target domain [48, 36, 41, 64].”
Few-Shot Adaptation of Generative Adversarial Networks (2020, methodology): “Sample-efﬁcient Image Synthesis Sample-efﬁcient image synthesis methods encourage diverse and high-quality generation in the low-data regime, most commonly through pretraining (Wang et al., 2018; Noguchi & Harada, 2019) or simultaneous training (Yamaguchi et al., 2019) on large image datasets.”
Few-Shot Incremental Learning for Label-to-Image Translation (2022, background): “Moreover, it is shown modulation in normalization benefits few-shot learning [14] and class information transfer [20].”
Towards Faster and Stabilized GAN Training for High-fidelity Few-shot Image Synthesis (2021, methodology): “Training from scratch vs. ﬁne-tuning : Fine-tuning from a pre-trained GAN (Mo et al., 2020; Noguchi & Harada, 2019; Wang et al., 2020) has been the go-to method for the image generation task on datasets with few samples.”
On Leveraging Pretrained GANs for Limited-Data Generation (2020, methodology): “The other special case of AdaFM is the Filter Selection (FS) presented in (Noguchi & Harada, 2019), which employs onerank simplification to both scale γ and shift β. Specifically, γ = γ̂1T and shift β = β̂1T with γ̂ ∈ RCout and β̂ ∈ RCout (see Figures 1(d) and 1(e)).”
One-Shot Generative Domain Adaptation (2021, methodology): “As our GenDA could capture the representative attributes precisely with an increasing number of target shots, we also Transferring GANs (TGAN) [14], Batch Statistics Adaptation (BSA) [9], FreezeD [8], MineGAN [13], EWC [7] and Cross-Domain [10].”
Multi-Concept Customization of Text-to-Image Diffusion (2022, background): “For example, one can adapt photorealistic faces into cartoons [18,36,44,47,48].”
Ablating Concepts in Text-to-Image Diffusion Models (2023, background): “Fine-tuning aims to adapt the weights of a pretrained generative model to new domains [73, 46, 72, 41, 79, 34, 47, 80, 30, 35, 24, 44], downstream tasks [71, 54, 78], and test images [6, 53, 48, 31, 25, 49].”
Revisiting Fine-tuning for Few-shot Learning (2019, methodology): “Although the methods proposed by Noguchi & Harada (2019) and Sun et al. (2019) presented similar ideas, we chose the former because of its simplicity in implementation.”
LFS-GAN: Lifelong Few-Shot Image Generation (2023, methodology): “For bias parameter, because of its single dimensionality, we do not apply weight decomposition.”
On Leveraging Pretrained GANs for Generation with Limited Data (2020, methodology): “Compared to transfer learning on discriminative tasks, fewer efforts have been made for generation tasks (Shin et al., 2016b; Wang et al., 2018b; Noguchi & Harada, 2019), as summarized in Section 2.”