Toward Multimodal Image-to-Image Translation

By Jun-Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexei A. Efros, Oliver Wang, Eli Shechtman. 2017

1,461 citations recorded by the source corpus.

Many image-to-image translation problems are ambiguous, as a single input image may correspond to multiple possible outputs. In this work, we aim to model a \emph{distribution} of possible outputs in a conditional generative modeling setting. The ambiguity of the mapping is distilled in a low-dimensional latent vector, which can be randomly sampled at test time. A generator learns to map the given input, combined with this latent code, to the output. We explicitly encourage the connection between output and the latent code to be invertible. This helps prevent a many-to-one mapping from the latent code to the output during training, also known as the problem of mode collapse, and produces more diverse results. We explore several variants of this approach by employing different training objectives, network architectures, and methods of injecting the latent code. Our proposed method encourages bijective consistency between the latent encoding and output modes. We present a systematic comparison of our method and other variants on both perceptual realism and diversity.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Large Scale Image Completion via Co-Modulated Generative Adversarial Networks (2021, background): “A branch of works aims to enforce the intra-conditioning diversity using VAE-based latent sampling strategies (Zhu et al., 2017b) or imposing distance-based loss terms (Huang et al., 2018; Mao et al., 2019; Qin et al., 2018).”
Learning Layout and Style Reconfigurable GANs for Controllable Image Synthesis (2020, methodology): “There are a few other methods tested at the resolution of 64×64 in the Layout2Im [20], including the pix2pixHD [14], BicycleGAN [58] and GauGAN [15], which are outperformed by the Layout2Im and thus not included here for the clarity of the table.”
Semantic Image Synthesis With Spatially-Adaptive Normalization (2019, background): “Furthermore, similarly to existing class-conditional generators [34,36,51], the new generator can take a random vector as input, enabling a sim ple and natural way for multi-modal synthesis [20,57].”
GAN Inversion: A Survey (2021, background): “CycleGAN [184], from the perspective of joint distribution matching.”
High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs (2017, methodology): “Second, these methods focus on global color and texture changes and allow no object-level control on the generated content [64].”
Multimodal Unsupervised Image-to-Image Translation (2018, methodology): “The style reconstruction loss Li recon is reminiscent of the latent reconstruction loss used in the prior works [11, 31, 44, 57].”
Physics-Constrained Deep Learning for High-dimensional Surrogate Modeling and Uncertainty Quantification without Labeled Data (2019, background): “Mode collapse is a well-known problem for conditional GANs [80, 81] and VAEs [82], which seems not much of a concern for flow-based generative models as demonstrated with the diversity of samples.”
Learning Multimodal Graph-to-Graph Translation for Molecular Optimization (2018, background): “Image/Text Style Translation Our work is closely related to image-to-image translation (Isola et al., 2017), which was later extended by Zhu et al. (2017) to learn multimodal mappings.”
Lifelong GAN: Continual Learning for Conditional Image Generation (2019, methodology): “BicycleGAN is proposed to take advantage of both cy- cles, hence the objective function is: LBicycleGAN = min G,E max D LcVAE−GAN + LcLR−GAN.”
A Survey on Neural Speech Synthesis (2021, background): “2) Providing more variance information such as pitch, duration, and energy as decoder input, which eases the one-to-many mapping problem [136, 81, 376, 449] in text to speech12.”
FastSpeech 2: Fast and High-Quality End-to-End Text to Speech (2020, background): “TTS is a typical one-to-many mapping problem [4, 6, 29], since multiple possible speech sequences can correspond to a text sequence due to different variations in speech audio, such as pitch, duration, sound volume and prosody.”
StarGAN v2: Diverse Image Synthesis for Multiple Domains (2019, methodology): “nection between stochastic noise and the generated image for diversity, by marginal matching [1], latent regression [40, 13], and diversity regularization [35, 27].”
Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space? (2019, background): “In the few past years, the quality of images synthesized by GANs has increased rapidly.”
StyleFlow: Attribute-conditioned Exploration of StyleGAN-Generated Images using Conditional Continuous Normalizing Flows (2020, methodology): “2017), BicycleGAN (Zhu et al. 2017b), pix2pixHD (Wang et al.”
Encoding in Style: a StyleGAN Encoder for Image-to-Image Translation (2020, methodology): “While this requires specialized changes in standard image-to-image architectures [44, 14], our framework inherently supports this by simply sampling style vectors.”
Stochastic Adversarial Video Prediction (2018, methodology): “Although this overcomes the limitations of pixel-wise losses, GANs are notoriously susceptible to mode collapse, where latent random variables are often ignored by the model, especially in the conditional setting [7,8,9,10].”
Video-to-Video Synthesis (2018, methodology): “To achieve multimodal synthesis [19,73,83], we adopt a feature embedding scheme [73] for the source video that consists of instance-level semantic segmentation masks.”
Pluralistic Image Completion (2019, methodology): “Some recent work utilizing these in image translation can produce diverse output [47, 21], but in such situations the conditionto-sample mappings are more local (e.”