MineGAN: Effective Knowledge Transfer From GANs to Target Domains With Few Images

By Yaxing Wang, Abel Gonzalez-Garcia, David Berga, Luis Herranz, Fahad Shahbaz Khan, Joost van de Weijer. 2019

208 citations recorded by the source corpus.

One of the attractive characteristics of deep neural networks is their ability to transfer knowledge obtained in one domain to other related domains. As a result, high-quality networks can be trained in domains with relatively little training data. This property has been extensively studied for discriminative networks but has received significantly less attention for generative models. Given the often enormous effort required to train GANs, both computationally as well as in the dataset collection, the re-use of pretrained GANs is a desirable objective. We propose a novel knowledge transfer method for generative models based on mining the knowledge that is most beneficial to a specific target domain, either from a single or multiple pretrained GANs. This is done using a miner network that identifies which part of the generative distribution of each pretrained GAN outputs samples closest to the target domain. Mining effectively steers GAN sampling towards suitable regions of the latent space, which facilitates the posterior finetuning and avoids pathologies of other methods such as mode collapse and lack of flexibility. We perform experiments on several complex datasets using various GAN architectures (BigGAN, Progressive GAN) and show that the proposed method, called MineGAN, effectively transfers knowledge to domains with few target images, outperforming existing methods. In addition, MineGAN can successfully transfer knowledge from multiple pretrained GANs. Our code is available at: \url{https://github.com/yaxingwang/MineGAN}.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

StyleGAN-NADA (2021, methodology): “Others introduce new parameters to control channel-wise statistics [34], steer sampling towards suitable regions of the latent space [55], add regularization terms [28, 51] or enforce cross-domain consistency while adapting to a target style [35].”
Freeze the Discriminator: a Simple Baseline for Fine-Tuning GANs (2020, methodology): “Numerous techniques have been proposed to overcome this limitation, e.g ., transferring knowledge of a well-trained source model [45, 32, 44], learning meta-knowledge for quick adaptation to a target domain [24, 47, 42], using an auxiliary task to facilitate training [7, 26, 48, 49], improving an…”
StyleAlign: Analysis and Applications of Aligned StyleGAN Models (2021, background): “…ﬁne-tuning are enabling few-shot tasks such as classiﬁcation (Chen et al., 2019) and unconditional generation (Wang et al., 2018b; Mo et al., 2020; Wang et al., 2020; Li et al., 2020; Ojha et al., 2021), improved performance in a wide variety of tasks (Devlin et al., 2018; Radford et al., 2018;…”
Ablating Concepts in Text-to-Image Diffusion Models (2023, background): “Fine-tuning aims to adapt the weights of a pretrained generative model to new domains [73, 46, 72, 41, 79, 34, 47, 80, 30, 35, 24, 44], downstream tasks [71, 54, 78], and test images [6, 53, 48, 31, 25, 49].”
Modeling Image Composition for Complex Scene Generation (2022, background): “Recently, [7, 19, 25, 38] extend the few-shot generation to objects with similar structures, such as human faces, buildings and cars.”
On Leveraging Pretrained GANs for Generation with Limited Data (2020, methodology): “Accordingly, our proposed method, when compared to the fine-tuning methods (Wang et al., 2018b; 2020), is expected to suffer less from overfitting and behave more robustly in the presence of differences between the source and target domains; when compared to (Noguchi & Harada, 2019), our method is…”
Regularizing Generative Adversarial Networks under Limited Data (2021, methodology): “model [15], modifying the batch statics [12], using an additional mining network [14], and fine-tuning parts of the GAN model [11].”
When, Why, and Which Pretrained GANs Are Useful? (2022, background): “A line of works (Wang et al., 2018b; Noguchi & Harada, 2019; Zhao et al., 2020; Mo et al., 2020; Wang et al., 2020; Li et al., 2020) investigate different methods to transfer GANs to new datasets and report significant advantages compared to training from scratch both in terms of generative quality ”
One-Shot Generative Domain Adaptation (2021, methodology): “As our GenDA could capture the representative attributes precisely with an increasing number of target shots, we also Transferring GANs (TGAN) [14], Batch Statistics Adaptation (BSA) [9], FreezeD [8], MineGAN [13], EWC [7] and Cross-Domain [10].”
GAN Prior Embedded Network for Blind Face Restoration in the Wild (2021, methodology): “They further proposed a novel knowledge transfer method for generative models by using a knowledge mining network [45].”
DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation (2022, background): “[69] Yaxing Wang, Abel Gonzalez-Garcia, David Berga, Luis”
Multi-Concept Customization of Text-to-Image Diffusion (2022, methodology): “A method of efficiently producing a whole distribution of images is leveraging a pretrained model and then using transfer learning [18,21,36,39,44,46–48,74, 75,82].”
Few-shot Image Generation via Cross-domain Correspondence (2021, background): “[8] Yaxing Wang, Abel Gonzalez-Garcia, David Berga, Luis Herranz, Fahad Shahbaz Khan, and Joost van de Weijer.”
Towards Faster and Stabilized GAN Training for High-fidelity Few-shot Image Synthesis (2021, methodology): “Training from scratch vs. ﬁne-tuning : Fine-tuning from a pre-trained GAN (Mo et al., 2020; Noguchi & Harada, 2019; Wang et al., 2020) has been the go-to method for the image generation task on datasets with few samples.”
Few-shot Image Generation with Elastic Weight Consolidation (2020, methodology): “Both [28] and [41] also focus on adapting models from a source to a target domain but they introduce additional parameters.”