The Degrees of Freedom of Partial Least Squares Regression

By Nicole Krämer, Masashi Sugiyama. 2010

111 citations recorded by the source corpus.

The derivation of statistical properties for partial least squares regression can be a challenging task. The reason is that the construction of latent components from the predictor variables also depends on the response variable. While this typically leads to good performance and interpretable models in practice, it makes the statistical analysis more involved. In this work, we study the intrinsic complexity of partial least squares regression. Our contribution is an unbiased estimate of its degrees of freedom. It is defined as the trace of the first derivative of the fitted values, seen as a function of the response. We establish two equivalent representations that rely on the close connection of partial least squares to matrix decompositions and Krylov subspace techniques. We show that the degrees of freedom depend on the collinearity of the predictor variables: The lower the collinearity, the higher the complexity. In particular, they are typically higher than the naive approach that defines the degrees of freedom as the number of components. Further, we illustrate how our degrees of freedom estimate can be used for the comparison of different regression methods. In the experimental section, we show that our degrees of freedom estimate in combination with information criteria is useful for model selection.

Subfields: Analytical Chemistry, Computer Vision and Pattern Recognition, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/3f0859500958f7788f9d2930db9f4161b7d1eb65

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Analytical Chemistry

In Computer Vision and Pattern Recognition

In Statistics and Probability

Citations in the corpus

Methodology and theory for partial least squares applied to functional data (2012, background): “1 include those of Phatak, Reilly and Penlidis (2002), Phatak and de Hoog (2003), Bro and Eldén (2009) and Krämer and Sugiyama (2011).”
mixOmics: An R package for ‘omics feature selection and multiple data integration (2017, background): “Components between different data sets however are not linearly dependent as we maximise the covariance between them [46].”
The Three-Pass Regression Filter: A New Approach to Forecasting Using Many Predictors (2015, unknown): “Kramer and Sugiyama (2011) provide an unbiased degrees of freedom calculation and associated information criterion for PLS forecasting problems.”
Statistical Computing in Functional Data Analysis: The R Package fda.usc (2012, methodology): “pls() where the PLS factors have been calculated using an alternative formulation of the NIPALS algorithm proposed by Krämer and Sugiyama (2011). The call for tecator example and the print of the fitted object are shown below:”
Functional Principal Component Regression and Functional Partial Least‐squares Regression: An Overview and a Comparative Study (2017, methodology): “The usual definition of the degrees of freedom parameter in regression (see, Krämer and Sugiyama, 2011) can not be applied for the best subset functional principal component regression and functional partial least squares regression estimation methods because the corresponding hat matrices depends…”