Unsupervised Cross-Domain Image Generation

By Yaniv Taigman, Adam Polyak, Lior Wolf. 2016

1,046 citations recorded by the source corpus.

We study the problem of transferring a sample in one domain to an analog sample in another domain. Given two related domains, S and T, we would like to learn a generative function G that maps an input sample from S to the domain T, such that the output of a given function f, which accepts inputs in either domains, would remain unchanged. Other than the function f, the training data is unsupervised and consist of a set of samples from each domain. The Domain Transfer Network (DTN) we present employs a compound loss function that includes a multiclass GAN loss, an f-constancy component, and a regularizing component that encourages G to map samples from T to themselves. We apply our method to visual domains including digits and face images and demonstrate its ability to generate convincing novel images of previously unseen entities, while preserving their identity.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Domain randomization for transferring deep neural networks from simulation to the real world (2017, background): “…the model in the target domain (e.g., [52]), adapting the weights of the model based on the statistics of the source and target domains (e.g., [22]), learning invariant features between domains (e.g., [47]), and learning a mapping from the target domain to the source domain (e.g., [43]).”
Generating Diverse High-Fidelity Images with VQ-VAE-2 (2019, background): “The samples generated from these models are hard to distinguish from real data without close inspection, and their applications range from super resolution [21] to domain editing [44], artistic manipulation [36], or text-to-speech and music generation [25].”
High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs (2017, background): “ as transforming Google maps to satellite views and generating cats from user sketches. Various methods have also been proposed to learn an image-to-image translation in the absence of training pairs [2,34,35,47,50,52,56,65]. Recently, Chen and Koltun [5] suggest that it might be hard for conditiona”
StarGAN: Unified Generative Adversarial Networks for Multi-domain Image-to-Image Translation (2017, methodology): “The idea of conditional image generation has also been successfully applied to domain transfer [9, 28], super-resolution imaging[14], and photo editing [2, 27].”
Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks (2017, methodology): “In particular, we adopt the technique of Taigman et al. [48] and regularize the generator to be near an identity mapping when real samples of the target domain are provided as the input to the generator: i.e.”
Unsupervised Image-to-Image Translation Networks (2017, background): “The results of the other algorithms were duplicated from (Taigman et al., 2017)”
Rewriting a Deep Generative Model (2020, background): “Recently, thanks to rapid advances of deep generative models [26,45,31], learning-based image synthesis and editing methods have become widely-used tools in the community, enabling applications such as manipulating the semantics of an input scene [60,6,72,16], image colorization [83,34,52,85], photo”
Self-Attention Generative Adversarial Networks (2018, background): “GANs have achieved great success in various image generation tasks, including image-to-image translation [9, 40, 29, 14], image super-resolution [12, 28] and text-to-image synthesis [24, 23, 37].”
StackGAN: Text to Photo-Realistic Image Synthesis with Stacked Generative Adversarial Networks (2016, background): “There is also work conditioned on images to generate images, including photo editing [2, 39], domain transfer [32, 12] and super-resolution [31, 15].”
Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization (2017, methodology): “GANs have also been applied to style transfer [31] and cross-domain image generation [50, 3, 23, 38, 37, 25].”
StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networks (2017, background): “There are also works conditioned on images to generate images, including photo editing [4], [57], domain transfer [18], [43] and super-resolution [23], [42].”
Toward Controlled Generation of Text (2017, background): “Dosovitskiy & Brox (2016); Taigman et al. (2017) use discriminators to measure high-level perceptual similarity.”
Adversarial Discriminative Domain Adaptation (2017, background): “GANs have also been explicitly applied to domain transfer tasks, such as domain transfer networks [18], which seek to directly map source images into target images.”
Multimodal Unsupervised Image-to-Image Translation (2018, background): “Some works enforce the translation to preserve certain properties of the source domain data, such as pixel values [21], pixel gradients [22], semantic features [10], class labels [22], or pairwise sample distances [16].”
Style Transfer from Non-Parallel Text by Cross-Alignment (2017, background): “Style transfer in vision Non-parallel style transfer has been extensively studied in computer vision (Gatys et al., 2016; Zhu et al., 2017; Liu and Tuzel, 2016; Liu et al., 2017; Taigman et al., 2016; Kim et al., 2017; Yi et al., 2017).”
Learning to Discover Cross-Domain Relations with Generative Adversarial Networks (2017, background): “Similarly, Taigman et al. (2016) translated face images to emojis by providing image features from pre-trained face recognition module as conditional input to a GAN.”
Algorithms and Theory for Multiple-Source Adaptation (2018, methodology): “We train the ConvNet (or CNN) architecture following Taigman et al. [2017] as our source models and joint model.”
Black-Box vs. White-Box: Understanding Their Advantages and Weaknesses From a Practical Point of View (2019, background): “face frontal view generation [50], generating new human poses [51], photos to emojis [52], photograph editing [53]–[55], face aging [56], [57], photo blending [58], super resolution [59]–[61], photo inpainting [62]–[64], video prediction [65], and 3D object generation [66], [67]; among”
Multi-Level Variational Autoencoder: Learning Disentangled Representations from Grouped Observations (2017, methodology): “…translation between two unpaired images collections using GAN-based architectures, see Zhu et al. [2017], Kim et al. [2017], Yi et al. [2017], Fu et al. [2017], Taigman et al. [2017], Shrivastava et al. [2017], Bousmalis et al. [2016], while Liu et al. [2017] employ a combination of VAE and GANs.”