Transferring GANs: generating images from limited data

By Yaxing Wang, Chenshen Wu, Luis Herranz, Joost van de Weijer, Abel Gonzalez-Garcia, B. Raducanu. 2018

320 citations recorded by the source corpus.

Transferring the knowledge of pretrained networks to new domains by means of finetuning is a widely used practice for applications based on discriminative models. To the best of our knowledge this practice has not been studied within the context of generative deep networks. Therefore, we study domain adaptation applied to image generation with generative adversarial networks. We evaluate several aspects of domain adaptation, including the impact of target domain size, the relative distance between source and target domain, and the initialization of conditional GANs. Our results show that using knowledge from pretrained networks can shorten the convergence time and can significantly improve the quality of the generated images, especially when the target data is limited. We show that these conclusions can also be drawn for conditional GANs even when the pretrained model was trained without conditioning. Our results also suggest that density may be more important than diversity and a dataset with one or few densely sampled classes may be a better source model than more diverse datasets such as ImageNet or Places.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

StyleGAN-NADA (2021, background): “[35], MineGAN [55], TGAN [56] and TGAN + ADA [20].”
GAN Memory with No Forgetting (2020, methodology): “, CelebA [43]; often a dense dataset is preferred [85]) and (ii) it’s sufficiently trained and shows relatively high generation quality.”
Memory Replay GANs: learning to generate images from new categories without forgetting (2018, background): “This has also been observed in GANs [24, 27] (shown later in Figures 3, 5 and 7 in the experiments section).”
Semantics Disentangling for Text-To-Image Generation (2019, background): “GAN-based algorithms show excellent performance in image generation [21, 1, 20, 25, 35, 2, 23].”
Freeze the Discriminator: a Simple Baseline for Fine-Tuning GANs (2020, background): “Numerous techniques have been proposed to overcome this limitation, e.g ., transferring knowledge of a well-trained source model [45, 32, 44], learning meta-knowledge for quick adaptation to a target domain [24, 47, 42], using an auxiliary task to facilitate training [7, 26, 48, 49], improving an…”
Image Generation From Small Datasets via Batch Statistics Adaptation (2019, methodology): “However, especially in image generation [34], 1,000 training examples are still necessary.”
LFS-GAN: Lifelong Few-Shot Image Generation (2023, methodology): “We established our work on the baseline (i) TGAN [45].”
When, Why, and Which Pretrained GANs Are Useful? (2022, methodology): “A line of works (Wang et al., 2018b; Noguchi & Harada, 2019; Zhao et al., 2020; Mo et al., 2020; Wang et al., 2020; Li et al., 2020) investigate different methods to transfer GANs to new datasets and report significant advantages compared to training from scratch both in terms of generative quality…”
Recent advances and clinical applications of deep learning in medical image analysis (2021, methodology): “Given a slice group, a non-local operation based module (Wang et al., 2018a) was employed to seize long-range dependencies of different positions and different channels in the feature map.”
MineGAN: Effective Knowledge Transfer From GANs to Target Domains With Few Images (2019, methodology): “Only recently,Wang et al. [33] studied ﬁnetun-ing from a single pretrained generative model and showed that it is beneﬁcial for domains with scarce data.”
StyleAlign: Analysis and Applications of Aligned StyleGAN Models (2021, background): “…unconditional generation (Wang et al., 2018b; Mo et al., 2020; Wang et al., 2020; Li et al., 2020; Ojha et al., 2021), improved performance in a wide variety of tasks (Devlin et al., 2018; Radford et al., 2018; He et al., 2020) and faster training convergence (Wang et al., 2018b; He et al., 2019).”
Few-shot Image Generation with Elastic Weight Consolidation (2020, background): “With a large number of examples available, both training from scratch and ﬁne-tuning from a pretrained model could lead to a good generative model for the target domain, while the ﬁne-tuning simply gets faster convergence [42].”
On Leveraging Pretrained GANs for Limited-Data Generation (2020, methodology): “Accordingly our method, when compared to TransferGAN, is expected to suffer less from overfitting and behave more robustly towards the differences between source and target domains; when compared to (Noguchi & Harada, 2019), our method is more flexible and thus provides clearly better generation, be”
Unbalanced GANs: Pre-training the Generator of Generative Adversarial Network using Variational Autoencoder (2020, background): “Adopting these, Wang et al. (2018) studied about transferring pre-trained knowledge on GANs and proposed Transferring GANs.”
On Data Augmentation for GAN Training (2020, background): “It is along with recent observations [14], [3], [15], [19], the more data GAN model trains, the higher quality it gets.”
Few-shot Image Generation via Cross-domain Correspondence (2021, background): “Note that this result is in alignment with the one presented in TaDensity Coverage TGAN [9] 0.”