Diverse Image-to-Image Translation via Disentangled Representations

By Hsin-Ying Lee, Hung-Yu Tseng, Jia-Bin Huang, Maneesh Kumar Singh, Ming-Hsuan Yang. 2018

1,049 citations recorded by the source corpus.

Image-to-image translation aims to learn the mapping between two visual domains. There are two main challenges for many applications: (1) the lack of aligned training pairs and (2) multiple possible outputs from a single input image. In this work, we present an approach based on disentangled representation for producing diverse outputs without paired training images. To achieve diversity, we propose to embed images onto two spaces: a domain-invariant content space capturing shared information across domains and a domain-specific attribute space. Using the disentangled features as inputs greatly reduces mode collapse. To handle unpaired training data, we introduce a novel cross-cycle consistency loss. Qualitative results show that our model can generate diverse and realistic images on a wide range of tasks. We validate the effectiveness of our approach through extensive evaluation.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

StarGAN v2: Diverse Image Synthesis for Multiple Domains (2019, methodology): “Here, MUNIT and DRIT suffer from mode-collapse in AFHQ, which results in lower LPIPS and higher FID than other methods.”
Few-Shot Unsupervised Image-to-Image Translation (2019, background): “Several works use the shared/partially-shared latent space assumption [29, 30]/[19, 26].”
Unsupervised Discovery of Disentangled Manifolds in GANs (2020, methodology): “Generative adversarial networks (GANs) [1,6,7,8,9] have been widely used for the image generation task.”
UNIT-DDPM: UNpaired Image Translation with Denoising Diffusion Probabilistic Models (2021, methodology): “Our method outperforms the contemporary approaches of CycleGAN [44], UNIT [25], MUNIT [16], and DRIT++ [24] over all of the benchmark datasets Facade, Photos–Maps, Summer–Winter, and RGB–Thermal offering a signiﬁcant average increase in performance of ∼ 20% against previous approaches across all…”
A Survey on Generative Adversarial Networks: Variants, Applications, and Training (2020, background): “ image into style and content codes and then recombined content code in the target style space with a random style code. Diverse I2I Translation via Disentangled Representations Image-to-Image (DRIT) [119] model is a multi-model generation framework that can generate different results with un-paired”
Network-to-Network Translation with Conditional Invertible Neural Networks (2020, methodology): “In the case of image conditioning, our approach can be trained either with aligned image pairs as in [33, 7, 51] or with unaligned image pairs as in [87, 39, 9, 32, 21].”
DeceptionNet: Network-Driven Domain Randomization (2019, background): “Nonetheless, we believe that both DRIT and PixelDA are not fully reachable by target-agnostic methods like ours since the space of all needed adaptations (such as aberrations or JPEG artifacts) has to be spanned by our deception modules.”
InstaGAN: Instance-aware Image-to-Image Translation (2018, background): “In particular, the unpaired (or unsupervised) image-to-image translation has achieved an impressive progress based on variants of generative adversarial networks (GANs) (Zhu et al., 2017; Liu et al., 2017; Choi et al., 2017; Almahairi et al., 2018; Huang et al., 2018; Lee et al., 2018), and has also”
A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications (2020, background): “Multi-modal image-to-image translation related works can be found in [272]–[275].”
RobustNet: Improving Domain Generalization in Urban-Scene Segmentation via Instance Selective Whitening (2021, methodology): “GDWCT applies this loss to image-to-image translation for more signiﬁcant style changes than other methods [23, 25] of aligning only the ﬁrst-order statistics ( i.e., channel-wise mean and variance).”
Feature Quantization Improves GAN Training (2020, methodology): “The test datasets consist of 120 (horse), 140 (zebra), 751 (photo), and 400 (vangogh). cat2dog and photo2portrait These datasets are used in DRIT (Lee et al., 2018).”
Mimicry: Towards the Reproducibility of GAN Research (2020, background): “In particular, GANs are known for their creative applications in areas such as image synthesis [36, 17, 32, 18] image-to-image translation [42, 43, 23, 24], video-to-video synthesis [39, 38, 7], and even 3D view synthesis [29, 30, 40].”
Residual Tuning: Toward Novel Category Discovery Without Labels (2022, neutral): “The work in [59] proposed a disentangled representation framework by embedding images onto a domain-invariant content space and a domain-speciﬁc attribute space.”
Joint Discriminative and Generative Learning for Person Re-Identification (2019, methodology): “Following the common practice in image-to-image translations [16,21,61], we use a large weight λ img = 5 for the image reconstruction loss.”
Image Generation From Layout (2018, methodology): “In-foGAN [2], DrNet [5] and DRIT [21] learn the disen-tangled representations in an unsupervised manner, either by maximizing the mutual information [2] or adversarial losses [5, 21].”
Learning Blind Video Temporal Consistency (2018, background): “Recent advances of deep convolutional neural networks (CNNs) have led to the development of many powerful image processing techniques including, image ﬁltering [30,37], enhancement [10,24,38], style transfer [17,23,29], colorization [19,41], and general image-to-image translation tasks [21,27,43].”