Domain Adaptation with Conditional Transferable Components

By Mingming Gong, Kun Zhang, Tongliang Liu, Dacheng Tao, Clark Glymour, Bernhard Schölkopf. 2016

374 citations recorded by the source corpus.

Domain adaptation arises in supervised learning when the training (source domain) and test (target domain) data have different distributions. Let X and Y denote the features and target, respectively, previous work on domain adaptation mainly considers the covariate shift situation where the distribution of the features P(X) changes across domains while the conditional distribution P(Y∣X) stays the same. To reduce domain discrepancy, recent methods try to find invariant components T(X) that have similar P(T(X)) on different domains by explicitly minimizing a distribution discrepancy measure. However, it is not clear if P(Y∣T(X)) in different domains is also similar when P(Y∣X) changes. Furthermore, transferable components do not necessarily have to be invariant. If the change in some components is identifiable, we can make use of such components for prediction in the target domain. In this paper, we focus on the case where P(X∣Y) and P(Y) both change in a causal system in which Y is the cause for X. Under appropriate assumptions, we aim to extract conditional transferable components whose conditional distribution P(T(X)∣Y) is invariant after proper location-scale (LS) transformations, and identify how P(Y) changes between domains simultaneously. We provide theoretical analysis and empirical evaluation on both synthetic and real-world data to show the effectiveness of our method.

Subfields: Representation & Self-Supervised Learning, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

In Meta & Continual Learning

Citations in the corpus

Toward Causal Representation Learning (2021, background): “It also led to a range of follow-up work [32], [78], [97], [114], [115], [152], [153], [156], [167], [195], [204], [243], [263], [267], [277], [278], [281], complementing the studies of Bareinboim and Pearl [14], [185], and it inspired a thread of work in the statistics community exploiting invarian”
Causality for Machine Learning (2019, background): “The line of work provided rather fruitful (Zhang et al., 2013; Weichwald et al., 2014; Zhang et al., 2015; Gong et al., 2016; Huang et al., 2017; Blöbaum et al., 2016; Guo et al., 2018; Wang et al., 2019; Subbaswamy et al., 2018; Rabanser et al., 2018; Li et al., 2018a; Lipton et al., 2018; Li et al”
Counterfactual Normalization: Proactively Addressing Dataset Shift and Improving Reliability Using Causal Mechanisms (2018, methodology): “These methods work by re-weighting the training data or extracting transferable features (e.g., Shimodaira (2000); Gretton et al. (2009); Gong et al. (2016); Zhang et al. (2013)).”
Domain Generalization via Conditional Invariant Representations (2018, methodology): “(Gong et al. 2016) proposed to learn components whose conditional distribution P(h(X)|Y ) is invariant across domains and estimate the target label distribution P(Y ) through labeled source domain data and unlabeled target domain data.”
Conditional variance penalties and domain shift robustness (2017, methodology): “The variable can be assumed to be latent in the setting of Gong et al. (2016).”
Domain Adaptation by Using Causal Inference to Predict Invariant Conditional Distributions (2017, methodology): “The methods proposed in [Schölkopf et al., 2012, Zhang et al., 2013, 2015, Gong et al., 2016] all address challenging settings in which conditional independences that follow from the usual Markov and faithfulness assumptions alone do not sufﬁce to solve the problem, but additional assumptions on…”
Deep Transfer Learning with Joint Adaptation Networks (2016, background): “Previous shallow transfer learning methods bridge the source and target domains by learning invariant feature representations or estimating the instance importance without using target labels [1, 2, 3, 4, 5, 6].”
Interventional Few-Shot Learning (2020, background): “6 [41] Gong Mingming, Zhang Kun, Liu Tongliang, Tao Dacheng, Clark Glymour, and Bernhard Schölkopf.”
Causal Transfer Learning (2017, methodology): “The methods proposed in [Schölkopf et al., 2012, Zhang et al., 2013, 2015, Gong et al., 2016] all address challenging settings in which conditional independences that follow from the usual Markov and faithfulness assumptions alone do not sufﬁce to solve the problem, but additional assumptions on…”
Learning Semantic Representations for Unsupervised Domain Adaptation (2018, methodology): “These methods aim at matching the marginal distributions across domains while (Zhang et al., 2013; Gong et al., 2016) considers the conditional distribution shift problem.”
Stratified Transfer Learning for Cross-domain Activity Recognition (2017, background): “[10] proposed transfer kernel learning (TKL), which learned a domain-invariant kernel in RKHS. [35] studied the conditional transfer components between domains.”
Knowledge Amalgamation from Heterogeneous Networks by Common Feature Learning (2019, background): “Domain adaptation [Ben-David et al., 2010; Gong et al., 2016], which belongs to transfer learning, aims at improving the testing performance on an unlabeled target domain while the model is trained on a related yet different source domain.”
Deep Domain Generalization via Conditional Invariant Adversarial Networks (2018, background): “As shown in [23, 24], we may be able to determine the conditional invariant representation T (X) by matching the marginal distribution P (T (X)) across domains, i.”
Joint distribution optimal transportation for domain adaptation (2017, methodology): “Wifi localization regression dataset For the regression task, we use the cross-domain indoor Wifi localization dataset that was proposed by Zhang and co-authors [4], and recently studied in [5].”
On Localized Discrepancy for Domain Adaptation (2020, background): “Domain adaptation problem is analyzed as structured generalization models from the causal view Zhang et al. (2013); Gong et al. (2016).”
Support and Invertibility in Domain-Invariant Representations (2019, background): “However, (Gong et al. , 2016) Next, we argue that that searching for a representation Z such that p s ( Z ) ≈ p t ( Z ) is often undesirable, and that learning objectives in the style of (6) are insensitive to information lost in domain-invariant representations.”
Causal Generative Domain Adaptation Networks (2018, background): “(Zhang et al., 2013; Gong et al., 2016) assumed simple location-scale transforms of PX|Y and found the identifiability conditions.”