Exemplar Guided Unsupervised Image-to-Image Translation

By Liqian Ma, Xu Jia, Stamatios Georgoulis, Tinne Tuytelaars, Luc Van Gool. 2018

146 citations recorded by the source corpus.

Image-to-image translation has recently received significant attention due to advances in deep learning. Most works focus on learning either a one-to-one mapping in an unsupervised way or a many-to-many mapping in a supervised way. However, a more practical setting is many-to-many mapping in an unsupervised way, which is harder due to the lack of supervision and the complex inner- and cross-domain variations. To alleviate these issues, we propose the Exemplar Guided & Semantically Consistent Image-to-image Translation (EGSC-IT) network which conditions the translation process on an exemplar image in the target domain. We assume that an image comprises of a content component which is shared across domains, and a style component specific to each domain. Under the guidance of an exemplar from the target domain we apply Adaptive Instance Normalization to the shared content component, which allows us to transfer the style information of the target domain to the source domain. To avoid semantic inconsistencies during translation that naturally appear due to the large inner- and cross-domain variations, we introduce the concept of feature masks that provide coarse semantic guidance without requiring the use of any semantic labels. Experimental results on various datasets show that EGSC-IT does not only translate the source image to diverse instances in the target domain, but also preserves the semantic consistency during the process.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Object-Centric Image Generation from Layouts (2020, methodology): “Masked object embeddings (Ma et al. 2018; Sun and Wu 2019) and bounding-box layouts (using 1-hot embeddings) have been previously used in the layout to image setting.”
Diverse Image-to-Image Translation via Disentangled Representations (2018, background): “Very recently, several concurrent works [1,17,5,28] (all independently developed) also adopt a disentangled representation similar to our work for learning diverse I2I translation from unpaired training data.”
Applications of Generative Adversarial Networks (GANs): An Updated Review (2019, methodology): “[74] proposed the Exemplar Guided and Semantically Consistent Image-to-image Translation (EGSC-IT) network which conditions the translation process on an exemplar image in the target domain.”
DRIT++: Diverse Image-to-Image Translation via Disentangled Representations (2019, methodology): “We note several concurrent methods (Almahairi et al. 2018; Cao et al. 2018; Huang et al. 2018; Lin et al. 2018a, b; Ma et al. 2018) (all independently developed) also adopt disentangled representations similar to our work for learning diverse I2I translation from unpaired training data.”
Discover, Hallucinate, and Adapt: Open Compound Domain Adaptation for Semantic Segmentation (2021, methodology): “Motivated by the previous works [15, 18, 26, 14, 4, 35] that utilizes style information as domain-specific representation, we propose to use latent target styles to cluster the compound target.”
RobustNet: Improving Domain Generalization in Urban-Scene Segmentation via Instance Selective Whitening (2021, methodology): “Semantic segmentation in DG Based on the synthetic data such as GTAV [47] and SYNTHIA [49], numerous DA studies [43, 60, 53, 6, 69, 18, 58, 37, 65] have been proposed in semantic segmentation, but only a few DG studies [64, 44] address semantic segmentation, as the majority of the DG methods mainly…”
Progressive Pose Attention Transfer for Person Image Generation (2019, background): “Generative Adversarial Networks (GAN) [6], which are basically composed of generator and discriminator that are trained in an adversarial way, can usually generate sharp images [6, 25, 30, 13, 17, 28, 12, 48, 20, 42].”
Cross-Domain Correspondence Learning for Exemplar-Based Image Translation (2020, methodology): “Baselines We compare our method with state-of-the-art image translation methods: 1) Pix2pixHD [45], a leading supervised approach; 2) SPADE [38], a recently proposed supervised translation method which also supports the style injection from an exemplar image; 3) MUNIT [17], an unsupervised method th”
Self-Ensembling With GAN-Based Data Augmentation for Domain Adaptation in Semantic Segmentation (2019, methodology): “In particular, concurrent works [21, 31] employ an adaptive instance normalization [20] to transfer style from the exemplar to the original image.”
Enhancing Photorealism Enhancement (2021, background): “transfer and example-based approaches [24] is the dependence on a favorable reference image.”