Multimodal Unsupervised Image-to-Image Translation

By Xun Huang, Ming-Yu Liu, Serge J. Belongie, Jan Kautz. 2018

2,739 citations recorded by the source corpus.

Unsupervised image-to-image translation is an important and challenging problem in computer vision. Given an image in the source domain, the goal is to learn the conditional distribution of corresponding images in the target domain, without seeing any pairs of corresponding images. While this conditional distribution is inherently multimodal, existing approaches make an overly simplified assumption, modeling it as a deterministic one-to-one mapping. As a result, they fail to generate diverse outputs from a given source domain image. To address this limitation, we propose a Multimodal Unsupervised Image-to-image Translation (MUNIT) framework. We assume that the image representation can be decomposed into a content code that is domain-invariant, and a style code that captures domain-specific properties. To translate an image to another domain, we recombine its content code with a random style code sampled from the style space of the target domain. We analyze the proposed framework and establish several theoretical results. Extensive experiments with comparisons to the state-of-the-art approaches further demonstrates the advantage of the proposed framework. Moreover, our framework allows users to control the style of translation outputs by providing an example style image. Code and pretrained models are available at this https URL

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

A Style-Based Generator Architecture for Generative Adversarial Networks (2018, methodology): “We choose to reuse the word “style” for y because similar network architectures are already used for feedforward style transfer [26], unsupervised image-toimage translation [27], and domain mixtures [22].”
On Buggy Resizing Libraries and Surprising Subtleties in FID Calculation (2021, methodology): “In particular, in Figure 9 (left) we evaluate the different intermediate checkpoints when training an image-to-image translation model [32] on the horse2zebra dataset.”
Large Scale Image Completion via Co-Modulated Generative Adversarial Networks (2021, background): “The unpaired setting is also investigated when paired training data is not available (Choi et al., 2018; Huang et al., 2018; Kim et al., 2019; Lazarow et al., 2017; Liu et al., 2017; Yi et al., 2017; Zhao et al., 2020b; Zhu et al., 2017a).”
Learning Layout and Style Reconfigurable GANs for Controllable Image Synthesis (2020, background): “, class information [3], [5], [6], [9], [36], source image [10], [12], [13], [37], text description [16], [17], [38], [39], scene graph [18], [40], etc) as input.”
Semantic Image Synthesis With Spatially-Adaptive Normalization (2019, methodology): “Conditional normalization layers include the Conditional Batch Normalization (Conditional BN) [12] and Adaptive Instance Normalization (AdaIN) [19], Both were first used in the style transfer task and later adopted in various vision tasks [5,9,11,20,25,34,36,39,46,51].”
InstructPix2Pix: Learning to Follow Image Editing Instructions (2022, neutral): “Generative models for image editing Image editing models traditionally targeted a single editing task such as style transfer [15, 16] or translation between image domains [22, 24, 36, 42, 71].”
Rewriting a Deep Generative Model (2020, methodology): “…have become widely-used tools in the community, enabling applications such as manipulating the semantics of an input scene [60,6,72,16], image colorization [83,34,52,85], photo stylization [25,38,56,54], image-to-image translation [35,86,9,73,55,32], and face editing and synthesis [24,58,63].”
Dynamic Neural Networks: A Survey (2021, neutral): “I & S Image segmentation [32], [119], [136], [138], [140], [144], [146], [193], [194], [195], [196], [197], Image-to-image translation [198], Object detection [105], [106], [137], [138], [153], Semantic image synthesis [199], [200], [201], Computer Image denoising [202], Fine-grained classification ”
High Fidelity Speech Synthesis with Adversarial Networks (2019, neutral): “Similar approaches have also been used in image-to-image transfer (Huang et al., 2018) and video synthesis (Saito & Saito, 2018).”
StarGAN v2: Diverse Image Synthesis for Multiple Domains (2019, methodology): “Here, MUNIT and DRIT suffer from mode-collapse in AFHQ, which results in lower LPIPS and higher FID than other methods.”
Encoding in Style: a StyleGAN Encoder for Image-to-Image Translation (2020, background): “While this requires specialized changes in standard image-to-image architectures [44, 14], our framework inherently supports this by simply sampling style vectors.”
Adversarial Latent Autoencoders (2020, background): “Given the information flow between E and G, the architecture is effectively mimicking a multiscale style transfer from E to G, with the difference that there is not an extra input image that provides the content [21, 22].”
Video-to-Video Synthesis (2018, background): “To the best of our knowledge, a general-purpose solution to video-to-video synthesis has not yet been explored by prior work, although its image counterpart, the image-to-image translation problem, is a popular research topic [6,31,33,43,44,63,66,73,82,83].”
Seeing What a GAN Cannot Generate (2019, background): “Generative Adversarial Networks [15] have enabled many computer vision and graphics applications such as generation [7, 21, 22], image and video manipulation [19, 20, 30, 35, 39, 41, 52], object recognition [6, 42], and textto-image translation [33, 45, 49].”
Image Synthesis From Reconfigurable Layout and Style (2019, background): “class information [25, 24, 35], source image [19, 39, 13], text description [29, 36], etc).”
Few-Shot Unsupervised Image-to-Image Translation (2019, methodology): “We also train several state-of-the-art two-domain translation models including CycleGAN [51], UNIT [26], and MUNIT [18].”
Diverse Image Generation via Self-Conditioned GANs (2020, methodology): “Due to its rapid advance, GANs have been used in a wide range of computer vision and graphics applications [40, 18, 41, 45, 17, 16].”
DLOW: Domain Flow for Adaptation and Generalization (2018, methodology): “A group of recent works aim to reduce the distribution difference on the feature level by learning domain-invariant features [13, 18, 31, 16], while others work on the image level to transfer source images to mimic the target domain style [64, 38, 65, 27, 1, 8].”
Embracing Imperfect Datasets: A Review of Deep Learning Solutions for Medical Image Segmentation (2019, methodology): “The authors use a pix2pix network Huang et al. (2018a) for image synthesis, which transforms an image of a healthy CXR into one with pathology. They observe that across different segmentation models, this augmentation significantly increases precision, recall, and Dice score. Shin et al. (2018) use ”