Kernel ridge vs. principal component regression: Minimax bounds and the qualification of regularization operators
By Lee H. Dicker, Dean Phillips Foster, Daniel J. Hsu. 2017
83 citations recorded by the source corpus.
: Regularization is an essential element of virtually all kernel methods for nonparametric regression problems. A critical factor in the effectiveness of a given kernel method is the type of regularization that is employed. This article compares and contrasts members from a general class of regularization techniques, which notably includes ridge regression and principal component regression. We derive an explicit finite-sample risk bound for regularization-based estimators that simultaneously accounts for (i) the structure of the ambient function space, (ii) the regularity of the true regression function, and (iii) the adaptability (or qualification ) of the regularization. A simple consequence of this upper bound is that the risk of the regularization-based estimators matches the minimax rate in a vari- ety of settings. The general bound also illustrates how some regularization techniques are more adaptable than others to favorable regularity prop- erties that the true regression function may possess. This, in particular, demonstrates a striking difference between kernel ridge regression and ker- nel principal component regression. Our theoretical results are supported by numerical experiments.
Subfields: Computational Mechanics, Mathematical Physics, Statistics and Probability
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computational Mechanics
In Mathematical Physics
In Statistics and Probability
Citations in the corpus
- Frequentist coverage and sup-norm convergence rate in Gaussian process regression (2017, background): “For past work on risk bounds for the KRR estimator in other norms, refer to [19, 29, 41, 52, 54].”