Wasserstein Distance Guided Representation Learning for Domain Adaptation

By Jian Shen, Yanru Qu, Weinan Zhang, Yong Yu. 2017

1,029 citations recorded by the source corpus.

Domain adaptation aims at generalizing a high-performance learner on a target domain via utilizing the knowledge distilled from a source domain which has a different but related data distribution. One solution to domain adaptation is to learn domain invariant feature representations while the learned representations should also be discriminative in prediction. To learn such representations, domain adaptation frameworks usually include a domain invariant representation learning approach to measure and reduce the domain discrepancy, as well as a discriminator for classification. Inspired by Wasserstein GAN, in this paper we propose a novel approach to learn domain invariant feature representations, namely Wasserstein Distance Guided Representation Learning (WDGRL). WDGRL utilizes a neural network, denoted by the domain critic, to estimate empirical Wasserstein distance between the source and target samples and optimizes the feature extractor network to minimize the estimated Wasserstein distance in an adversarial manner. The theoretical advantages of Wasserstein distance for domain adaptation lie in its gradient property and promising generalization bound. Empirical studies on common sentiment and image classification adaptation datasets demonstrate that our proposed WDGRL outperforms the state-of-the-art domain invariant representation learning approaches.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Learngene: From Open-World to Your Learning Task (2021, background): “Transfer Learning Methods such as (Huang et al. 2006; Sugiyama et al. 2008; Sun et al. 2011; Shen et al. 2018; Lee et al. 2019; Duan et al. 2009; Zhuang et al. 2009; Ev-geniou, Micchelli, and Pontil 2005), which focus on transferring the knowledge across domains, aim at improving the performance of…”
Domain Adaptation with Asymmetrically-Relaxed Distribution Alignment (2019, background): “Standard choices of D such as a domain classiﬁer (Jensen-Shannon (JS) divergence 2 ) (Ganin et al., 2016), Wasserstein distance (Shen et al., 2018) or Maximum Mean Discrepancy (Huang et al., 2007) have the property that D ( p , p ) = 0 if p ≡ p and D ( p , p ) > 0 otherwise.”
WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020, neutral): “For example, methods that learn features that have similar distributions across domains are equally applicable to both settings (Ben-David et al., 2006; Long et al., 2015; Sun et al., 2016; Ganin et al., 2016; Tzeng et al., 2017; Shen et al., 2018; Wu et al., 2019).”
UDALM: Unsupervised Domain Adaptation through Language Modeling (2021, methodology): “For example, according to Shen et al. (2018) when a domain clas-siﬁer can perfectly distinguish target from source representations, there will be a gradient vanishing problem.”
CoDA: Contrast-enhanced and Diversity-promoting Data Augmentation for Natural Language Understanding (2020, methodology): “We adopt the multi-kernel MMD implementation based on Shen et al. (2018)5, to quantify the distance of data distributions before and after DA transformations.”
Deep Visual Domain Adaptation: A Survey (2018, methodology): “hrough playing a minimax game against features from S. Rather than using discriminator to classify domain label, some papers make some other explorations. Inspired by Wasserstein GAN [1], Shen et al. [102] utilized discriminator to estimate empirical Wasserstein distance between the source and targe”
Learning to Map Natural Language Instructions to Physical Quadcopter Control using Simulated Flight (2019, methodology): “The Wasserstein distance estimation procedure from Shen et al. [40] requires a discriminator that is K-Lipschitz continuous.”
DeepJDOT: Deep Joint distribution optimal transport for unsupervised domain adaptation (2018, methodology): “[28] used the Wasserstein distance as a loss in a deep learning setting to promote similarities between embedded representations using the dual formulation of the problem exposed in [29].”
On Learning Invariant Representations for Domain Adaptation (2019, methodology): “, 2014; 2015; 2016), the Wasserstein distance (Courty et al., 2017b;a; Shen et al., 2018; Lee & Raginsky, 2018), etc.”
A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications (2020, background): “Other transfer learning based GANs can be found in [52], [256]–[262].”
A Holistic Approach to Undesired Content Detection in the Real World (2022, positive): “We follow Shen et al. (2018) and approximate the Wasserstein distance by maximizing the loss of a domain critic head.”
Attending to Discriminative Certainty for Domain Adaptation (2019, methodology): “Wasserstein distance based discriminator was used in [39] to bring the two distributions closer.”
Neural Temporality Adaptation for Document Classification: Diachronic Word Embeddings and Domain Adaptation Models (2019, methodology): “Finally, we use the Wasser-stein distance (Shen et al., 2018) to measure the differences across temporal domains.”
Domain-Specific Batch Normalization for Unsupervised Domain Adaptation (2019, methodology): “Global alignment techniques include Maximum Mean Discrepancy (MMD) [1, 28], Central Moment Discrepancy (CMD) [29], Wassterstein distance [21], and CORrelation ALignment (CORAL) [22], some of which are reformulated using deep neural networks [12, 23, 28].”
Reliable Weighted Optimal Transport for Unsupervised Domain Adaptation (2020, methodology): “Wasserstein Distance Guided Representation Learning (WDGRL) [34] uses Wasserstein distance as a core loss in promoting similarities between embedded representations by the dual formulation of the problem.”
Overcoming Negative Transfer: A Survey (2020, background): “For example, Ganin et al. [9] developed the domain adversarial neural network (DANN) with a binary domain discriminator, Shen et al. [65] proposed a Wasserstein distance based domain discriminator.”
Robust Optimal Transport with Applications in Generative Modeling and Domain Adaptation (2020, methodology): “Commonly used distance measures include Wasserstein distance [24] and non-saturating loss [10].”