Analysis of Kernel Mean Matching under Covariate Shift

By Yaoliang Yu, Csaba Szepesvari. 2012

97 citations recorded by the source corpus.

In real supervised learning scenarios, it is not uncommon that the training and test sample follow different probability distributions, thus rendering the necessity to correct the sampling bias. Focusing on a particular covariate shift problem, we derive high probability confidence bounds for the kernel mean matching (KMM) estimator, whose convergence rate turns out to depend on some regularity measure of the regression function and also on some capacity measure of the kernel. By comparing KMM with the natural plug-in estimator, we establish the superiority of the former hence provide concrete evidence/ understanding to the effectiveness of KMM under covariate shift.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Domain Adaptation with Conditional Transferable Components (2016, methodology): “Theoretical analysis of importance reweighting methods for correcting covariate shift has also been studied in (Cortes et al., 2010; Yu & Szepesv´ari, 2012).”
Domain Adaptation with Asymmetrically-Relaxed Distribution Alignment (2019, background): “Traditional approaches to both problems require the source distributions’ support to cover the target support, estimating adapted classiﬁers via importance-weighted risk minimization (Shimodaira, 2000; Huang et al., 2007; Gretton et al., 2009; Yu & Szepesv´ari, 2012; Lipton et al., 2018).”
Characterizing and Avoiding Negative Transfer (2018, methodology): “In our approach, a discriminator estimating both marginal and joint distributions is used as a gate to filter potentially harmful source data by reducing the bias between source and target risks, which corresponds to the idea of importance reweighting [5, 37].”
Stable Prediction across Unknown Environments (2018, methodology): “A variety of techniques have been proposed to estimate the density ratio, including discriminative estimation [2], Kullaback-Leibler importance estimation [3], kernel mean matching [4] [19], maximum entropy methods [5], minimax optimization [20], and robust biasaware approach [6].”
A Review of Domain Adaptation without Target Labels (2019, methodology): “The non-parametric alternative is to use kernel density estimators [93], [94], [95]:”
Overcoming Negative Transfer: A Survey (2020, background): “In this scenario, leveraging the knowledge from the best candidate may be effective against negative transfer as long as the best source knowledge is sufﬁciently related [24], [32], [34], [47].”
A Survey on Negative Transfer (2020, background): “When there are multiple source domains, selectively aggregating the most similar ones to the target domain, or a weighted aggregation of all source domains, may achieve better performance than simply averaging all source domains [80], [81].”
A Review of Single-Source Deep Unsupervised Visual Domain Adaptation (2020, background): “While most of the work we discussed so far focuses on learning domain-invariant representations, methods based on estimating the importance ratio of density functions between target and source domains are also abundant in the literature [28, 186–189].”
Toward Minimax Off-policy Value Estimation (2015, background): “We also discuss the implications of these results for off-policy evaluation problems in contextual bandits and fixed-horizon Markov decision processes.”
Transfer String Kernel for Cross-Context DNA-Protein Binding Prediction (2016, methodology): “This problem has been investigated in both statistics and machine learning under various assumptions [8].”
An introduction to domain adaptation and transfer learning (2018, methodology): “Depending on if the weights are upper bounded, convergence rates for KMM can be computed [87,86,210]”
Causal Generative Domain Adaptation Networks (2018, background): “In the covariate shift scenario, where P X changes but P Y | X stays the same, to correct the shift in P X , traditional approaches reweight the source domain data by density ratios of the features [3, 5, 4, 13, 14].”
Smooth Operators (2013, background): “In Huang et al. (2007); Gretton et al. (2009); Yu & Szepesvari (2012) a quadratic program is given to estimate a weight vector β with non-negative entries , such that the following cost function is min- imised, m j=1 k(y j , ·) − n i=1 β i k(x i , ·) k .”
Covariate Shift in Hilbert Space: A Solution via Sorrogate Kernels (2013, methodology): “A theoretical analysis was given by (Yu & Szepesvári, 2012).”
Maximum Mean Discrepancy for Class Ratio Estimation: Convergence Bounds and Kernel Selection (2014, methodology): “The analysis of MMD for the covariance shift problem is very different (Yu & Szepesvari, 2012; Gretton et al., 2009; Cortes et al., 2008) from ours because their objectives and assumptions are different.”
Domain adaptation and sample bias correction theory and algorithm for regression (2014, methodology): “Yu and Szepesvári recently presented an analysis of the KMM estimator [37].”
Domain Generalization by Marginal Transfer Learning (2017, background): “For example, several works adopt the covariate shift assumption, which requires the source and target domains to have the same posterior, allowing the marginals to differ arbitrarily (Zadrozny, 2004; Huang et al., 2007; Cortes et al., 2008; Sugiyama et al., 2008; Bickel et al., 2009; Kanamori et al.”