Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks

By Konstantinos Bousmalis, Nathan Silberman, David Dohan, Dumitru Erhan, Dilip Krishnan. 2016

1,626 citations recorded by the source corpus.

Collecting well-annotated image datasets to train modern machine learning algorithms is prohibitively expensive for many tasks. One appealing alternative is rendering synthetic data where ground-truth annotations are generated automatically. Unfortunately, models trained purely on rendered images fail to generalize to real images. To address this shortcoming, prior work introduced unsupervised domain adaptation algorithms that have tried to either map representations between the two domains, or learn to extract features that are domain-invariant. In this work, we approach the problem in a new light by learning in an unsupervised manner a transformation in the pixel space from one domain to the other. Our generative adversarial network (GAN)-based method adapts source-domain images to appear as if drawn from the target domain. Our approach not only produces plausible samples, but also outperforms the state-of-the-art on a number of unsupervised domain adaptation scenarios by large margins. Finally, we demonstrate that the adaptation process generalizes to object classes unseen during training.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Learning Robust Global Representations by Penalizing Local Predictive Power (2019, background): “, 2018) and coercing domain invariance via adversarial learning (or further extensions) (Ganin et al., 2016; Bousmalis et al., 2017; Tzeng et al., 2017; Xie et al., 2018; Hoffman et al., 2018; Long et al., 2018; Zhao et al., 2018b; Kumar et al., 2018; Li et al., 2018b; Zhao et al., 2018a; Schoenauer”
High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs (2017, background): “Various methods have also been proposed to learn an image-to-image translation in the absence of training pairs [2, 33, 34, 46, 49, 51, 54, 63].”
Rewriting a Deep Generative Model (2020, background): “Recently, thanks to rapid advances of deep generative models [26,45,31], learning-based image synthesis and editing methods have become widely-used tools in the community, enabling applications such as manipulating the semantics of an input scene [60,6,72,16], image colorization [83,34,52,85], photo”
Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization (2017, methodology): “GANs have also been applied to style transfer [31] and cross-domain image generation [50, 3, 23, 38, 37, 25].”
Multimodal Unsupervised Image-to-Image Translation (2018, background): “Some works enforce the translation to preserve certain properties of the source domain data, such as pixel values [21], pixel gradients [22], semantic features [10], class labels [22], or pairwise sample distances [16].”
Visual Feature Attribution Using Wasserstein GANs (2017, methodology): “Generative adversarial images conditioned on an input image have been used in diverse applications such as video frame prediction [35], image super-resolution [33], imagetranslation across domains using paired [26] and unpaired [66] images, and pixel level domain adaptation [7, 49].”
Learning Independent Causal Mechanisms (2017, background): “Another research direction that is relevant to our work is unsupervised domain adaptation (Bousmalis et al., 2017).”
Deep Reinforcement Learning for Autonomous Driving: A Survey (2020, background): “In [121], the model learns a transformation in the pixel space from one domain to the other, in an unsupervised way.”
Learning to cluster in order to Transfer across domains and tasks (2017, background): “…of work dealing with domain shift between image datasets by minimizing domain discrepancy (Tzeng et al., 2014; Long et al., 2015b; Ganin et al., 2016; Sun et al., 2016; Long et al., 2016; Sener et al., 2016; Carlucci et al., 2017; Long et al., 2017; Zellinger et al., 2017; Bousmalis et al., 2017).”
Unsupervised Model Adaptation for Continual Semantic Segmentation (2020, methodology): “A large group of the existing UDA algorithms for image segmentation use adversarial learning for extracting domaininvariant features (Luc et al. 2016; Bousmalis et al. 2017; Hoffman et al. 2018b; Murez et al. 2018; Saito et al. 2018; Sankaranarayanan et al. 2018; Dhouib, Redko, and Lartizien 2020).”
Learning to Drive in a Day (2018, background): “Two areas that could greatly improve the availability of data for the application of reinforcement learning to real autonomous driving are semi-supervised learning [36] and domain transfer [37].”
Multi-Level Variational Autoencoder: Learning Disentangled Representations from Grouped Observations (2017, background): “Multiple works perform image-toimage translation between two unpaired images sets using adversarial training (Zhu et al. 2017; Kim et al. 2017; Yi et al. 2017; Fu et al. 2017; Taigman, Polyak, and Wolf 2017; Shrivastava et al. 2017; Bousmalis et al. 2017; Liu, Breuel, and Kautz 2017).”
CyCADA: Cycle-Consistent Adversarial Domain Adaptation (2017, methodology): “Bousmalis et al. (2017b) use a content similarity loss to ensure the generated target image is similar to the original source image; however, this requires prior knowledge about which parts of the image stay the same across domains (e.g. foreground).”
Delete, Retrieve, Generate: a Simple Approach to Sentiment and Style Transfer (2018, background): “These models are also difficult to train (Salimans et al., 2016; Arjovsky and Bottou, 2017; Bousmalis et al., 2017).”
Video-to-Video Synthesis (2018, background): “To the best of our knowledge, a general-purpose solution to video-to-video synthesis has not yet been explored by prior work, although its image counterpart, the image-to-image translation problem, is a popular research topic [6,31,33,43,44,63,66,73,82,83].”
Is Generator Conditioning Causally Related to GAN Performance? (2018, background): “, 2017), domain adaptation (Bousmalis et al., 2016), and super-resolution (Ledig et al.”
Semantics Disentangling for Text-To-Image Generation (2019, background): “The rapid progress of the Generative Adversarial Networks (GAN) [11, 21, 1, 20] brings a remarkable evolution in natural image generation with diverse conditions.”