Kernel dimension reduction in regression

By Kenji Fukumizu, Francis R. Bach, Michael I. Jordan. 2009

334 citations recorded by the source corpus.

We present a new methodology for sufficient dimension reduction (SDR). Our methodology derives directly from the formulation of SDR in terms of the conditional independence of the covariate X from the response Y , given the projection of X on the central subspace [cf. J. Amer. Statist. Assoc. 86 (1991) 316–342 and Regression Graphics (1998) Wiley]. We show that this conditional independence assertion can be characterized in terms of conditional covariance operators on reproducing kernel Hilbert spaces and we show how this characterization leads to an M-estimator for the central subspace. The resulting estimator is shown to be consistent under weak conditions; in particular, we do not have to impose linearity or ellipticity conditions of the kinds that are generally invoked for SDR methods. We also present empirical results showing that the new methodology is competitive in practice.

Subfields: Control and Systems Engineering, Statistics and Probability, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Control and Systems Engineering

In Statistics and Probability

In Statistical Learning Theory

Citations in the corpus

Kernel Mean Embedding of Distributions: A Review and Beyonds (2016, methodology): “For instance, Gretton et al. (2005a) uses a Hilbert-Schmidt norm of the cross-covariance operator as a measure of statistical dependence between two random variables; see also Chwialkowski and Gretton (2014) and Chwialkowski et al.”
Computing functions of random variables via reproducing kernel Hilbert space representations (2015, neutral): “For the notions of characteristic/universal kernels, see Steinwart (2002); Fukumizu et al. (2008, 2009); an example thereof is the Gaussian kernel (26) The kernel map of a distribution p can be deﬁned as the expectation of the feature map (Berlinet and Agnan 2004; Smola et al. 2007; Gretton et al.…”
Nonparametric Independence Testing for Small Sample Sizes (2014, background): “performance. 1 Introduction Reproducing Kernel Hilbert Spaces (RKHSs) have become an indispensable part of machine learning. The long list of applications includes kernel PCA [1], dimension reduction [2], structure discovery [3], independence testing [4], Granger causality in time series [5], learni”
Characteristic Kernels and Infinitely Divisible Distributions (2014, methodology): “The conditions are explored according to the type of domain X and the class of kernel (Gretton et al., 2007; Fukumizu et al., 2008, 2009a,b; Sriperumbudur et al., 2010).”
Density Estimation in Infinite Dimensional Exponential Families (2013, methodology): “The mean and covariance operators have found many applications in nonparametric hypothesis testing [20, 18] and dimensionality reduction [16, 17].”
Conditional mean embeddings as regressors (2012, methodology): “Conditional mean embeddings have been successfully applied to inference in graphical models, reinforcement learning, subspace selection, and conditional independence testing (Fukumizu et al., 2008; 2009; Song et al., 2009; 2010; Gr ¨ unew ¨ alder et al., 2012).”
Universality, Characteristic Kernels and RKHS Embedding of Measures (2010, methodology): “Recently, this idea has been generalized to embed probability distributions into RKHSs, which provides a linear method for dealing with higher order statistics (Gretton et al., 2007; Smola et al., 2007; Fukumizu et al., 2008, 2009b; Sriperumbudur et al., 2008, 2009a,b).”
Hilbert Space Embeddings and Metrics on Probability Measures (2009, methodology): “Fukumizu et al. (2008, 2009a) extended this characterization to non-compact M and showed that H is characteristic if and only if the direct sum of H and R is dense in the Banach space of r-integrable (for some r ≥ 1) functions.”
Joint covariate selection and joint subspace selection for multiple classification problems (2010, background): “There has been a large literature on sufficient dimension reduction (e.g., Chiaromonte and Cook 2002; Fukumizu et al. 2008; Li 1991), but the focus has been on univariate response variables.”
Emergence of Invariance and Disentanglement in Deep Representations (2017, background): “groupstructure,availableframeworkscouldnotquantifythesetradeoﬀsorprecisely predictobservedphenomena. Suﬃcientreduction Anapparentlyrelatedliteraturestreamisconcernedwithsuﬃcientdimensionalityreduction[9,12]where,however,therepresentationmapsahigh-dimensionaldataspacetoalowerdimensionalspace,typicall”
Minimax risks for sparse regressions: Ultra-high-dimensional phenomenons (2010, neutral): “Dimension reduction methods are often studied in more general settings than linear regression [17, 26].”
Gradient-Based Kernel Dimension Reduction for Regression (2014, methodology): “For KDR, the number of iterations for optimization is 150, and the bandwidth parameters are reduced during the iterations, as described in Fukumizu et al. (2009).”
Characteristic Kernels on Groups and Semigroups (2008, background): “defined by the element in H such that hmX;fiH = E[f(X)] (8f 2 H) (See [6, 7 ]).”
Kernel Bayes' rule: Bayesian inference with positive definite kernels (2013, background): “With an appropriate choice of positive definite kernel, the kernel mean on the RKHS uniquely determines the distribution of the variable (Fukumizu et al., 2004, 2009a; Sriperumbudur et al., 2010), and statistical inference problems on distributions can be solved via operations on the kernel means.”