Few-Shot Unsupervised Image-to-Image Translation

By Ming-Yu Liu, Xun Huang, Arun Mallya, Tero Karras, Timo Aila, Jaakko Lehtinen, Jan Kautz. 2019

679 citations recorded by the source corpus.

Unsupervised image-to-image translation methods learn to map images in a given class to an analogous image in a different class, drawing on unstructured (non-registered) datasets of images. While remarkably successful, current methods require access to many images in both source and destination classes at training time. We argue this greatly limits their use. Drawing inspiration from the human capability of picking up the essence of a novel object from a small number of examples and generalizing from there, we seek a few-shot, unsupervised image-to-image translation algorithm that works on previously unseen target classes that are specified, at test time, only by a few example images. Our model achieves this few-shot generation capability by coupling an adversarial training scheme with a novel network design. Through extensive experimental validation and comparisons to several baseline methods on benchmark datasets, we verify the effectiveness of the proposed framework. Our implementation and datasets are available at https://github.com/NVlabs/FUNIT

Subfields: Computer Vision and Pattern Recognition, Media Technology, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

In Generative Models

Citations in the corpus

On Buggy Resizing Libraries and Surprising Subtleties in FID Calculation (2021, background): “A wide range of image and video synthesis applications [45, 54, 67, 83] have been enabled, as a result of tremendous progress in deep generative models such as GANs [9, 28, 34, 37, 59], VAEs [41, 55, 61], autoregressive models [52], flow-based models [18, 40], and energy-based models [21, 50, 63].”
Large Scale Image Completion via Co-Modulated Generative Adversarial Networks (2021, background): “Although some multi-modal unpaired image-to-image translation methods propose to encode the style from another reference image (Huang et al., 2018; Liu et al., 2019b), this unreasonably assumes that the style representations are entirely independent of the conditional input and hence compromises…”
Learning Layout and Style Reconfigurable GANs for Controllable Image Synthesis (2020, neutral): “, class information [3], [5], [6], [9], [36], source image [10], [12], [13], [37], text description [16], [17], [38], [39], scene graph [18], [40], etc) as input.”
InstructPix2Pix: Learning to Follow Image Editing Instructions (2022, neutral): “Generative models for image editing Image editing models traditionally targeted a single editing task such as style transfer [15, 16] or translation between image domains [22, 24, 37, 43, 72].”
StarGAN v2: Diverse Image Synthesis for Multiple Domains (2019, methodology): “To address the scalability, several studies have proposed a unified framework [2, 6, 14, 24].”
Palette: Image-to-Image Diffusion Models (2021, methodology): “, 2018], few shot translation [Liu et al., 2019] and many more.”
A Disentangling Invertible Interpretation Network for Explaining Latent Representations (2020, methodology): “The subsequent experiments use the following datasets: AnimalFaces [28], DeepFashion [29, 31], CelebA [30] MNIST[27], Cifar10 [25], and FashionMNIST [50].”
Freeze the Discriminator: a Simple Baseline for Fine-Tuning GANs (2020, neutral): “…e.g ., transferring knowledge of a well-trained source model [45, 32, 44], learning meta-knowledge for quick adaptation to a target domain [24, 47, 42], using an auxiliary task to facilitate training [7, 26, 48, 49], improving an inference procedure of suboptimal models [2, 39, 29, 38],…”
Few-shot Video-to-Video Synthesis (2019, methodology): “Recently, the FUNIT [31] was proposed for generating images of unseen domains via the adaptive instance normalization technique [19].”
SimSwap: An Efficient Framework For High Fidelity Face Swapping (2020, methodology): “To avoid such phenomena, we utilize the idea of adversarial training [10, 16, 21, 28] and use the Discriminator to distinguish results with apparent error.”
Bowtie Networks: Generative Modeling for Joint Few-Shot Recognition and Novel-View Synthesis (2020, background): “Discussion and Future Work: Our experimental evaluation has focused on fine-grained categories, mainly because state-of-the-art novel-view synthesis models still cannot address image generation for a wide spectrum of general images (Liu et al., 2019).”
Image-to-Image Translation: Methods and Applications (2021, methodology): “Inspired by the human learning strategy, few- and one-shot I2I algorithms [17], [81], [112], [80] have been proposed to translate from very few (or even one) in the limit unpaired training examples of the source and target domains.”
Dual Contrastive Loss and Attention for GANs (2021, background): “See more samples in the supplementary material. dataset [55], and 70K CLEVR [38] dataset which contains rendered images with random compositions of 3D shapes, uniform materials, uniform colors, point lighting, and a plain background.”
Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Models (2019, background): “…(Pandey and Dukkipati, 2017; Pu et al., 2016; Sohn et al., 2015; Yan et al., 2016), as well as generative adversarial network (GAN)-based models for cross domain image-to-image translation (Ledig et al., 2017; Li and Wand, 2016; Liu et al., 2019; Taigman et al., 2017; Wang and Gupta, 2016).”
Deep learning for deepfakes creation and detection: A survey (2019, methodology): “thub.com/shaoanlu/fewshot-face-translation-GAN - Use a pre-trained face recognition model to extract latent embeddings for GAN processing. - Incorporate semantic priors obtained by modules from FUNIT [39] and SPADE [40]. DeepFaceLab https://github.com/iperov/DeepFaceLab - Expand from the Faceswap me”
GANcraft: Unsupervised 3D Neural Rendering of Minecraft Worlds (2021, background): “This task can be performed in the supervised setting [22, 26, 35, 49, 61, 72], where example pairs of corresponding images are available, as well as the unsupervised setting [14, 21, 32, 33, 35, 53, 71], where only two sets of images are available.”
Network-to-Network Translation with Conditional Invertible Neural Networks (2020, result): “This stands in contrast to current inﬂuential domain transfer approaches [41, 51, 83] that require learning or ﬁnetuning existing domain representations to facilitate transfer between them.”
Effectively Unbiased FID and Inception Score and Where to Find Them (2019, methodology): “Deep Generative Models have been used successfully to generate hyperrealistic images [18, 19, 7], map images between domains in an unsupervised fashion [42, 23], and generate images from text [40, 39].”
StyleGAN2 Distillation for Feed-forward Image Manipulation (2020, background): “The meth-ods of unpaired image-to-image translation can be either single mode GANs [60,58,35,10] or multimodal GANs [61,23,32,33,36,11].”
StyleAlign: Analysis and Applications of Aligned StyleGAN Models (2021, background): “Viewed in this light, alignment-based image translation resembles several previous image translation approaches (Liu et al., 2017; Huang et al., 2018; Liu et al., 2019a), which are based on shared latent spaces.”