Few-shot Image Generation with Elastic Weight Consolidation

By Yijun Li, Richard Zhang, Jingwan Lu, Eli Shechtman. 2020

190 citations recorded by the source corpus.

Few-shot image generation seeks to generate more data of a given domain, with only few available training examples. As it is unreasonable to expect to fully infer the distribution from just a few observations (e.g., emojis), we seek to leverage a large, related source domain as pretraining (e.g., human faces). Thus, we wish to preserve the diversity of the source domain, while adapting to the appearance of the target. We adapt a pretrained model, without introducing any additional parameters, to the few examples of the target domain. Crucially, we regularize the changes of the weights during this adaptation, in order to best preserve the information of the source dataset, while fitting the target. We demonstrate the effectiveness of our algorithm by generating high-quality results of different target domains, including those with extremely few examples (e.g., <10). We also analyze the performance of our method with respect to some important factors, such as the number of examples and the dissimilarity between the source and target domain.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Media Technology, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Media Technology

In Representation & Self-Supervised Learning

Citations in the corpus

StyleGAN-NADA (2021, background): “Others introduce new parameters in order to control channelwise statistics [28], leverage a miner network in order to steer GAN sampling towards suitable regions of the latent space [45], add novel regularization terms to the loss function [22, 42] or mix contrastive learning ideas with patchlevel d”
When, Why, and Which Pretrained GANs Are Useful? (2022, background): “A line of works (Wang et al., 2018b; Noguchi & Harada, 2019; Zhao et al., 2020; Mo et al., 2020; Wang et al., 2020; Li et al., 2020) investigate different methods to transfer GANs to new datasets and report significant advantages compared to training from scratch both in terms of generative quality ”
Multi-Concept Customization of Text-to-Image Diffusion (2022, methodology): “[36], we analyze the change in parameters for each layer in the fine-tuned model on the target dataset with the loss in Eqn.”
DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation (2022, result): “There has been research on techniques to avoid these pit-falls [35, 40, 45, 53, 66], although, in contrast to our work, this line of work primarily seeks to generate images that re-semble the target distribution but has no requirement of subject preservation.”
LFS-GAN: Lifelong Few-Shot Image Generation (2023, methodology): “Due to the lack of concerns on diversity, the existing modulation-based methods [8, 42, 54] show inferior diversity performance compared to regularization-based [29, 36, 47, 55] methods.”
StyleAlign: Analysis and Applications of Aligned StyleGAN Models (2021, background): “CF has been also studied in the context of GANs (Liang et al., 2018; Li et al., 2020; Thanh-Tung & Tran, 2020).”
Ablating Concepts in Text-to-Image Diffusion Models (2023, background): “Fine-tuning aims to adapt the weights of a pretrained generative model to new domains [73, 46, 72, 41, 79, 34, 47, 80, 30, 35, 24, 44], downstream tasks [71, 54, 78], and test images [6, 53, 48, 31, 25, 49].”
Modeling Image Composition for Complex Scene Generation (2022, background): “Recently, [7, 19, 25, 38] extend the few-shot generation to objects with similar structures, such as human faces, buildings and cars.”
Few-shot Image Generation via Cross-domain Correspondence (2021, methodology): “Figure 4: Comparison of our method compared to EWC [5] and FreezeD [6] in 1-shot, 5-shot and 10-shot setting.”