DRIT++: Diverse Image-to-Image Translation via Disentangled Representations

By Hsin-Ying Lee, Hung-Yu Tseng, Jia-Bin Huang, Maneesh Kumar Singh, Ming-Hsuan Yang. 2019

804 citations recorded by the source corpus.

Image-to-image translation aims to learn the mapping between two visual domains. There are two main challenges for this task: (1) lack of aligned training pairs and (2) multiple possible outputs from a single input image. In this work, we present an approach based on disentangled representation for generating diverse outputs without paired training images. To synthesize diverse outputs, we propose to embed images onto two spaces: a domain-invariant content space capturing shared information across domains and a domain-specific attribute space. Our model takes the encoded content features extracted from a given input and attribute vectors sampled from the attribute space to synthesize diverse outputs at test time. To handle unpaired training data, we introduce a cross-cycle consistency loss based on disentangled representations. Qualitative results show that our model can generate diverse and realistic images on a wide range of tasks without paired training data. For quantitative evaluations, we measure realism with user study and Fréchet inception distance, and measure diversity with the perceptual distance metric, Jensen–Shannon divergence, and number of statistically-different bins.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Weakly Supervised Disentanglement with Guarantees (2019, background): “Learning such representations that align with the underlying factors of variation may be critical to the development of machine learning models that are explainable or human-controllable (Gilpin et al., 2018; Lee et al., 2019; Klys et al., 2018).”
Pluralistic Image Completion (2019, background): “Some recent work utilizing these in image translation can produce diverse output [47, 21], but in such situations the conditionto-sample mappings are more local (e.”
Unpaired Image-to-Image Translation via Latent Energy Transport (2020, methodology): “Thus, these schemes generally require a combination of generators, discriminators or style encoders [25, 4, 5, 44], and complicate engineered loss designs.”
A Survey on Generative Adversarial Networks: Variants, Applications, and Training (2020, methodology): “Likewise, DRIT++ [120] expands the DRIT by 1) integrating sample diversity mode-seeking maximum likelihood, and 2) generalizing the two-domain structure to tackle multi-domain I2I translation issues.”
Unsupervised Discovery of Disentangled Manifolds in GANs (2020, methodology): “Generative adversarial networks (GANs) [1,6,7,8,9] have been widely used for the image generation task.”
One-Step Diffusion with Distribution Matching Distillation (2023, background): “…them, GANs have shown unprecedented quality in realism [4, 26– 28, 30, 67], particularly when the GAN loss can be combined with task-speciﬁc, auxiliary regression losses to mitigate training instability, ranging from paired image translation [24, 54, 79, 89] to unpaired image editing [37, 55, 94].”
MixNMatch: Multifactor Disentanglement and Encoding for Conditional Image Generation (2019, background): “, one for appearance and the other for pose, do exist [16, 18, 23, 26, 40].”
Controllable Continuous Gaze Redirection (2020, background): “set as 0, which means that these secondary attributes remain unchanged during the generation. But we can also set �4 1 as any value in [0,1]to generate diverse results, which makesourmethodmulti-modal[20,38,47].Ontheonehand,the“O” branches describe the common attributes of the same person when perfo”
Enjoy Your Editing: Controllable GANs for Image Editing via Latent Space Navigation (2021, neutral): “, 2019), (d) DRIT++ (Lee et al., 2020), and (e) Ours.”
Cross-Domain Correspondence Learning for Exemplar-Based Image Translation (2020, background): “Since the mapping from one image domain to another is inherently multi-modal, following works promote the synthesis diversity by performing stochastic sampling from the latent space [53, 17, 24].”
GAN Prior Embedded Network for Blind Face Restoration in the Wild (2021, background): “Many following works, such as unsupervised learning [52], disentangled learning [26], few-shot learning [32], high resolution image synthesis [43], multi-domain translation [8], multimodal translation [53], have been proposed to extend cGAN to different scenarios.”
CrDoCo: Pixel-Level Domain Transfer With Cross-Domain Consistency (2019, background): “In imageto-image translation, enforcing cycle consistency allows the network to learn the mappings without paired data [51, 22].”
Image-to-Image Translation: Methods and Applications (2021, methodology): “with an existing cGANs framework, such as DRIT [76], to generate more diverse translated images.”
Mode Seeking Generative Adversarial Networks for Diverse Image Synthesis (2019, methodology): “To achieve multimodality, several recent attempts [10, 14, 32] introduce additional encoder networks and objective functions to impose a bijection constraint between the latent code space and the image space.”