Impossibility Theorems for Domain Adaptation

By Shai Ben-David, Tyler Lu, Teresa Luu, Dávid Pál. 2010

320 citations recorded by the source corpus.

Subfields: Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

Citations in the corpus

Learning Robust Global Representations by Penalizing Local Predictive Power (2019, background): “While some methods have justiﬁed domain-adversarial learning by appealing to theoretical bounds due to Ben-David et al. (2010a), the theory does not in fact guarantee generalization (recently shown by Johansson et al. (2019) and Wu et al. (2019)) and sometimes guarantees failure.”
Toward Causal Representation Learning (2021, background): “It is nontrivial (and, in some cases, provably hard [21]) to learn statistical models that are stable across training environments and generalize to novel testing environments [2], [5], [167], [187], [204] drawn from the same environment distribution.”
Towards Causal Representation Learning (2021, background): “environments E does not coincide with the set of possible environments PG, we have an additional estimation error that may be arbitrarily large in the worst case [5, 20].”
Weakly-Supervised Disentanglement Without Compromises (2020, background): “, 2017) under covariate shifts (Shimodaira, 2000; Quionero-Candela et al., 2009; Ben-David et al., 2010), fairer downstream predictions (Locatello et al.”
Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift (2018, background): “Owing to the impossibility of correcting for shift absent assumptions [3], these papers often assume either covariate shift q(x, y) = q(x)p(y|x) [15, 45, 49] or label shift q(x, y) = q(y)p(x|y) [7, 29, 38, 48, 56].”
Learning Robust Representations by Projecting Superficial Statistics Out (2018, background): “Other papers bound worst-case performance under bounded shifts as measured by divergence measures on the train v.s. test distributions (Ben-David et al., 2010a; Mansour et al., 2009; Hu et al., 2016).”
Domain Adaptation with Asymmetrically-Relaxed Distribution Alignment (2019, methodology): “Although Ganin et al. (2016) motivate their idea via theoretical results due to Ben-David et al. (2010a), the theory is insuﬃcient to justify their method.”
Causal Transfer in Machine Learning (2015, background): “Indeed, if the test task diﬀers signiﬁcantly from the source tasks, we may run into the problem of negative transfer [Pan and Yang, 2010] and DG becomes impossible [Ben-David et al., 2010].”
Adaptation Based on Generalized Discrepancy (2019, methodology): “This intuition was formalized by Ben-David et al. (2010) and Ben-David and Urner (2012) who showed that even in the favorable scenario where the source and target distribution admit the same support, a sample of size in the order of that of the support is needed in order to solve the domain…”
Stability and Hypothesis Transfer Learning (2013, background): “Previous work has addressed in detail the theory of DA and proposed algorithms that critically depend on optimal weighting parameters given by the theoretical analysis (Ben-David et al., 2010a;b; Mansour et al., 2009b; Cortes et al., 2008).”
Adversarial target-invariant representation learning for domain generalization (2019, background): “The work of Ben-David et al. (2010a) established the theoretical foundation for studying the cross-domain generalization properties for domain adaptation problems.”
Feature-based transfer learning with real-world applications (2010, background): “In transductive transfer learning, there are some research works focusing on the generalization bound when the training and test data distributions are different [16, 24, 15, 17].”
Domain adaptation and sample bias correction theory and algorithm for regression (2014, background): “In the absence of that guarantee, even with a zero discrepancy in general adaptation may not be successful (see [2] for simple examples of this situation).”
ST3D: Self-training for Unsupervised Domain Adaptation on 3D Object Detection (2021, unknown): “[26, 27] explore domain-invariant feature learning by minimizing Maximum Mean Discrepancy [1].”
Invariant Models for Causal Transfer Learning (2015, background): “Indeed, if the test task differs significantly from the source tasks, we may run into the problem of negative transfer (Pan and Yang, 2010) and DG becomes impossible (Ben-David et al., 2010).”
Larger Norm More Transferable: An Adaptive Feature Norm Approach for Unsupervised Domain Adaptation (2018, background): “DA occurs when the underlying distributions corresponding to the source and target domains in the shared label space are different but similar [2] to make sense the transfer.”
AutoDIAL: Automatic Domain Alignment Layers (2017, background): “This is motivated by an impossibility theorem [2], which intuitively states that no learner relying on the covariate shift hypothesis, i.”