Image-to-image translation for cross-domain disentanglement

By Abel Gonzalez-Garcia, Joost van de Weijer, Yoshua Bengio. 2018

260 citations recorded by the source corpus.

Deep image translation methods have recently shown excellent results, outputting high-quality images covering multiple modes of the data distribution. There has also been increased interest in disentangling the internal representations learned by deep methods to further improve their performance and achieve a finer control. In this paper, we bridge these two objectives and introduce the concept of cross-domain disentanglement. We aim to separate the internal representation into three parts. The shared part contains information for both domains. The exclusive parts, on the other hand, contain only factors of variation that are particular to each domain. We achieve this through bidirectional image translation based on Generative Adversarial Networks and cross-domain autoencoders, a novel network component. Our model offers multiple advantages. We can output diverse samples covering multiple modes of the distributions of both domains, perform domain-specific image transfer and interpolation, and cross-domain retrieval without the need of labeled data, only paired images. We compare our model to the state-of-the-art in multi-modal image translation and achieve better results for translation on challenging datasets as well as for cross-domain retrieval on realistic datasets.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Learning deep representations by mutual information estimation and maximization (2018, background): “, 2014; Brakel & Bengio, 2017), disentangled (Schmidhuber, 1992; Rifai et al., 2012; Bengio et al., 2013; Chen et al., 2018; Gonzalez-Garcia et al., 2018), or independently controllable (Thomas et al.”
Offline Reinforcement Learning with Fisher Divergence Critic Regularization (2021, background): “…first popularized in machine learning by Wasserstein generative-adversarial networks (Arjovsky et al., 2017), but also used in imitation learning (Kostrikov et al., 2018), cross-domain disentan- glement (Gonzalez-Garcia et al., 2018), and uncertainty estimation (van Amersfoort et al., 2020).”
Exemplar Guided Unsupervised Image-to-Image Translation (2018, methodology): “As shown in concurrent works (Gonzalez-Garcia et al., 2018; Huang et al., 2018; Lee et al., 2018), we assume that an image is composed of two disentangled representations.”
Synthetic Data for Deep Learning (2019, background): “Further developments include asymmetric training for unsupervised domain adaptation [514], DistanceGAN for one-sided domain mapping [52], co-regularized alignment [334], cross-domain autoencoders [206], multisource domain adversarial networks [705], continuous cross-domain translation [367], face re”
Hi-CMD: Hierarchical Cross-Modality Disentanglement for Visible-Infrared Person Re-Identification (2019, background): “Recently, considerable attention has been focused on learning disentangled representations in various fields [12, 7, 13, 24].”
Image-to-Image Translation: Methods and Applications (2021, background): “[129] disentangled the representation of two domains into three parts: the shared part containing common information”
Describe What to Change: A Text-guided Unsupervised Image-to-image Translation Approach (2020, background): “The basic assumption of DWC-GAN is that each image can be decomposed in a domain-invariant content space and a domainspecific attribute space [14, 20, 29].”
Self-Ensembling With GAN-Based Data Augmentation for Domain Adaptation in Semantic Segmentation (2019, background): “To produce multi-modal outputs, one possible approach injects noise vectors as additional inputs to the generator [54, 1, 11], but it could lead to the mode collapse problem.”
Learning Disentangled Representations with Reference-Based Variational Autoencoders (2019, background): “…as: temporal coherence in sequential data (Hsu et al., 2017; Denton et al., 2017; Villegas et al., 2017), pairs of aligned images obtained from different domains (Gonzalez-Garcia et al., 2018) or knowledge about the rendering process in computer graphics (Yang et al., 2015; Kulkarni et al., 2015).”
Guided Variational Autoencoder for Disentanglement Learning (2020, background): “Disentanglement learning [41, 48, 23, 1, 16, 26] recently becomes a popular topic in representation learning.”
Deep Sketch-Guided Cartoon Video Inbetweening (2021, background): “to translate the images across multiple domains by learning the domain-invariant representation from data [32], [33], [34], [35].”
Dual Adversarial Inference for Text-to-Image Synthesis (2019, methodology): “…of our proposed dual adversarial inference on the disentanglement of content and style, we ﬁrst validate our proposed method on a toy dataset: MNIST-CB [8], where we formulate the digit generation problem as a text-to-image synthesis problem by considering the digit identity as the text content.”
MixNMatch: Multifactor Disentanglement and Encoding for Conditional Image Generation (2019, unknown): “content) [19, 17, 24, 11, 26, 41].”