Unsupervised Image-to-Image Translation Networks

By Ming-Yu Liu, Thomas Breuel, Jan Kautz. 2017

2,962 citations recorded by the source corpus.

Unsupervised image-to-image translation aims at learning a joint distribution of images in different domains by using images from the marginal distributions in individual domains. Since there exists an infinite set of joint distributions that can arrive the given marginal distributions, one could infer nothing about the joint distribution from the marginal distributions without additional assumptions. To address the problem, we make a shared-latent space assumption and propose an unsupervised image-to-image translation framework based on Coupled GANs. We compare the proposed framework with competing approaches and present high quality image translation results on various challenging unsupervised image translation tasks, including street scene image translation, animal image translation, and face image translation. We also apply the proposed framework to domain adaptation and achieve state-of-the-art performance on benchmark datasets. Code and additional results are available in this https URL .

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Progressive Growing of GANs for Improved Quality, Stability, and Variation (2017, background): “…methods that produce novel samples from high-dimensional data distributions, such as images, are ﬁnding widespread use, for example in speech synthesis (van den Oord et al., 2016a), image-to-image translation (Zhu et al., 2017; Liu et al., 2017), and image inpainting (Iizuka et al., 2017).”
Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors (2022, background): “UNIT [38] projected two different domains into a shared latent space and used a per-domain decoder to re-synthesize images in the desired domain.”
Towards Language-Free Training for Text-to-Image Generation (2021, background): “GANs [11] have inspired many advances in image synthesis [18, 20, 23, 28, 32].”
Large Scale Image Completion via Co-Modulated Generative Adversarial Networks (2021, background): “The unpaired setting is also investigated when paired training data is not available (Choi et al., 2018; Huang et al., 2018; Kim et al., 2019; Lazarow et al., 2017; Liu et al., 2017; Yi et al., 2017; Zhao et al., 2020b; Zhu et al., 2017a).”
VAEBM: A Symbiosis between Variational Autoencoders and Energy-based Models (2020, background): “, 2016a) generation, distribution alignment across domains (Zhu et al., 2017; Liu et al., 2017; Tzeng et al., 2017) and semi-supervised learning (Kingma et al.”
Semantic Image Synthesis With Spatially-Adaptive Normalization (2019, background): “Another widely-used form is image-to-image translation [18,20,23, 27, 49, 50], where both input and output are images.”
High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs (2017, background): “ as transforming Google maps to satellite views and generating cats from user sketches. Various methods have also been proposed to learn an image-to-image translation in the absence of training pairs [2,34,35,47,50,52,56,65]. Recently, Chen and Koltun [5] suggest that it might be hard for conditiona”
StarGAN: Unified Generative Adversarial Networks for Multi-domain Image-to-Image Translation (2017, methodology): “To alleviate the problem of obtaining data pairs, unpaired image-to-image translation frameworks [8, 16, 32] have been proposed.”
Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks (2017, background): “[27] extends this framework with a combination of variational autoencoders [23] and generative adversarial networks.”
Rewriting a Deep Generative Model (2020, background): “Recently, thanks to rapid advances of deep generative models [26,45,31], learning-based image synthesis and editing methods have become widely-used tools in the community, enabling applications such as manipulating the semantics of an input scene [60,6,72,16], image colorization [83,34,52,85], photo”
Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization (2017, methodology): “GANs have also been applied to style transfer [31] and cross-domain image generation [50, 3, 23, 38, 37, 25].”
Multimodal Unsupervised Image-to-Image Translation (2018, methodology): “[15] propose the UNIT framework, which assumes a shared latent space such that corresponding images in two domains are mapped to the same latent code.”
Style Transfer from Non-Parallel Text by Cross-Alignment (2017, background): “Style transfer in vision Non-parallel style transfer has been extensively studied in computer vision (Gatys et al., 2016; Zhu et al., 2017; Liu and Tuzel, 2016; Liu et al., 2017; Taigman et al., 2016; Kim et al., 2017; Yi et al., 2017).”
Adversarial Examples: Attacks and Defenses for Deep Learning (2017, background): “GAN can be used to augment data samples, provide image superresolution [57], and image-to-image translation [58], [59].”
The Perception-Distortion Tradeoff (2017, background): “GAN-based image restoration Most recently, GAN-based methods have demonstrated unprecedented perceptual quality in super-resolution [3], [6], [9], [37], inpainting [7], [20], [38], compression [21], [39], [40], deblurring [41] and image-to-image translation [11], [22], [42].”
Moment Matching for Multi-Source Domain Adaptation (2018, background): “moment matching [47], adversarial domain confusion [40, 8, 38] and GAN-based alignment [50, 15, 23].”
DualGAN: Unsupervised Dual Learning for Image-to-Image Translation (2017, neutral): “, [2, 9], does not lead to effective general-purpose solutions as its applicability is taskdependent, leading to unnatural image translation results, as shown in comparative studies by CycleGAN [26].”