A risk comparison of ordinary least squares vs ridge regression
By Paramveer S. Dhillon, Dean Phillips Foster, Sham M. Kakade, Lyle Ungar. 2011
50 citations recorded by the source corpus.
We compare the risk of ridge regression to a simple variant of ordinary least squares, in which one simply projects the data onto a finite dimensional subspace (as specified by a principal component analysis) and then performs an ordinary (un-regularized) least squares regression in this subspace. This note shows that the risk of this ordinary least squares method (PCA-OLS) is within a constant factor (namely 4) of the risk of ridge regression (RR).
Subfields: Analytical Chemistry, Applied Mathematics, Statistics and Probability
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Analytical Chemistry
- Jan Peters
- Rajen Dinesh Shah
- Daniel J. Hsu
- Dean Phillips Foster
- Sivaraman Balakrishnan
- Ming Yuan
- Yi Lin
- Yuhong Yang
In Applied Mathematics
- Jan Peters
- Rajen Dinesh Shah
- Daniel J. Hsu
- Dean Phillips Foster
- Sivaraman Balakrishnan
- Lee H. Dicker
In Statistics and Probability
- Jan Peters
- Rajen Dinesh Shah
- Sivaraman Balakrishnan
- Daniel J. Hsu
- Dean Phillips Foster
- Lee H. Dicker
- Bernhard Schölkopf
- Peter Buhlmann
Citations in the corpus
- The hardness of conditional independence testing and the generalised covariance measure (2018, methodology): “t gives a bound on the prediction error of kernel ridge regression with xed design. The arguments are similar to those used in the analysis of regular ridge regression, see for example Dhillon et al. [4]. Lemma 30. Let z 1;:::;z n2Z(for some input space Z) be deterministic and suppose x i= f(z i) + ”
- Kernel ridge vs. principal component regression: Minimax bounds and the qualification of regularization operators (2017, methodology): “…rates for estimation over finite-dimensional subspaces (Proposition 1) rely on slightly different arguments, which are based on well-known eigenvalue perturbation results that have been adapted to handle Hilbert-Schmidt operators (e.g., the Davis-Kahan sinΘ theorem (Davis and Kahan, 1970)).”
- Finding and Leveraging Structure in Learning Problems (2012, background): “There is also a large literature with sharp characterization of risk bounds for linear regression [Hsu et al., 2011, Dhillon et al., 2011, Audibert and Catoni, 2010].”