Support and Invertibility in Domain-Invariant Representations

By Fredrik D. Johansson, David A. Sontag, Rajesh Ranganath. 2019

177 citations recorded by the source corpus.

Learning domain-invariant representations has become a popular approach to unsupervised domain adaptation and is often justified by invoking a particular suite of theoretical results. We argue that there are two significant flaws in such arguments. First, the results in question hold only for a fixed representation and do not account for information lost in non-invertible transformations. Second, domain invariance is often a far too strict requirement and does not always lead to consistent estimation, even under strong and favorable assumptions. In this work, we give generalization bounds for unsupervised domain adaptation that hold for any representation function by acknowledging the cost of non-invertibility. In addition, we show that penalizing distance between densities is often wasteful and propose a bound based on measuring the extent to which the support of the source domain covers the target domain. We perform experiments on well-known benchmarks that illustrate the short-comings of current standard practice.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Learning Robust Global Representations by Penalizing Local Predictive Power (2019, background): “While some methods have justiﬁed domain-adversarial learning by appealing to theoretical bounds due to Ben-David et al. (2010a), the theory does not in fact guarantee generalization (recently shown by Johansson et al. (2019) and Wu et al. (2019)) and sometimes guarantees failure.”
In Search of Lost Domain Generalization (2020, background): “Although popular, learning domain-invariant features has received some criticism (Zhao et al., 2019; Johansson et al., 2019).”
Evaluating Prediction-Time Batch Normalization for Robustness under Covariate Shift (2020, methodology): “In addition, these methods can become brittle when the training and test distributions are highly distinct [Johansson et al., 2019].”
Generalizing to Unseen Domains: A Survey on Domain Generalization (2021, background): “To address this, Ben-David et al. [24] developed another bound ([24], Thm. 2; [25], Thm. 1): where the H ∆ H -divergence is deﬁned as d H ∆ H ( P sX , P tX ) := sup h,h (cid:48) ∈H | s ( h, h (cid:48) ) − t ( h, h (cid:48) ) | , replacing the total variation d 1 to measure the distribution…”
Domain Generalization using Causal Matching (2020, background): “Among them, several works (Zhao et al., 2019; Johansson et al., 2019; Akuzawa et al., 2019) show that the class-conditional methods (Li et al., 2018c;d; Ghifary et al., 2016; Hu et al., 2019) are better than those that enforce marginal domain-invariance of features (Muandet et al., 2013; Ganin et…”
A Review of Single-Source Deep Unsupervised Visual Domain Adaptation (2020, background): “In a concurrent work, (author?) [185] also identified the insufficiency of learning domain-invariant representation for successful adaptation.”
The Risks of Invariant Risk Minimization (2020, background): “However, there is evidence that invariance may not be sufficient for domain adaptation (Zhao et al., 2019; Johansson et al., 2019).”
Causally motivated shortcut removal using auxiliary labels (2021, background): “Thus our work contributes to ongoing investigations about whether there is a trade-oﬀ between fari-ness and accuracy [Zhang et al., 2019, Calders et al., 2009, Johansson et al., 2019, Dutta et al., 2020, Zhao and Gordon, 2019].”
Invariant Risk Minimization Games (2020, background): “Domain adaptation is known to have serious limitations even when the marginal distribution of labels shift across environments [22, 23].”
Domain Aggregation Networks for Multi-Source Domain Adaptation (2019, background): “Recent analysis (Zhao et al., 2019; Johansson et al., 2019) show that solely focusing on learning domain invariant features can be problematic when the marginal label distributions on Y between source and target domains are significantly different.”
Adversarial target-invariant representation learning for domain generalization (2019, methodology): “Moreover, we remark that the effectiveness of traditional domain adaptation results on top of which we built our method have been discussed recently in (Johansson et al., 2019), in which case one alternative bound has been introduced such that λ is dropped and the H-divergence is replaced by a measu”
On Learning Invariant Representations for Domain Adaptation (2019, background): “In a concurrent work, Johansson et al. (2019) also identified the insufficiency of learning domain-invariant representation for successful adaptation.”
Out-of-Distribution Generalization via Risk Extrapolation (REx) (2020, background): “Recent works have identified fundamental shortcomings with this approach, however (Zhao et al., 2019; Johansson et al., 2019; Arjovsky et al., 2019; Wu et al., 2020).”
On Localized Discrepancy for Domain Adaptation (2020, background): “In addition, Germain et al. (2016); Johansson et al. (2019) derive upper bounds for target error with divergences defined with the density ratio.”