Ridge regression and asymptotic minimax estimation over spheres of growing dimension

By Lee H. Dicker. 2016

83 citations recorded by the source corpus.

We study asymptotic minimax problems for estimating a $d$-dimensional regression parameter over spheres of growing dimension ($d\to \infty$). Assuming that the data follows a linear model with Gaussian predictors and errors, we show that ridge regression is asymptotically minimax and derive new closed form expressions for its asymptotic risk under squared-error loss. The asymptotic risk of ridge regression is closely related to the Stieltjes transform of the Mar\v{c}enko-Pastur distribution and the spectral distribution of the predictors from the linear model. Adaptive ridge estimators are also proposed (which adapt to the unknown radius of the sphere) and connections with equivariant estimation are highlighted. Our results are mostly relevant for asymptotic settings where the number of observations, $n$, is proportional to the number of predictors, that is, $d/n\to\rho\in(0,\infty)$.

Subfields: Applied Mathematics, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/c41097fcfa4d1eaf181e1776f8847295ead16f16

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Applied Mathematics

In Statistics and Probability

Citations in the corpus

Exact minimax risk for linear least squares, and the lower tail of sample covariance matrices (2019, background): “Another line of work [25, 21, 23, 26, 22] considers the limiting behavior of regression procedures in the high-dimensional asymptotic regime where d, n tend to infinity at a proportional rate, with their ratio kept constant [35].”
High-Dimensional Asymptotics of Prediction: Ridge Regression and Classification (2015, background): “Important references include, among others, Bickel and Levina (2004), Dicker (2014), El Karoui (2013), Fujikoshi et al. (2011), Hsu et al. (2014), Saranadasa (1993), and Zollanvari and Dougherty (2015).”
Cleaning large correlation matrices: tools from random matrix theory (2016, background): “34) and unconditional estimator is not that surprising in the large N limit and has been mentioned in different contexts [130, 131].”
Kernel ridge vs. principal component regression: Minimax bounds and the qualification of regularization operators (2017, background): “On the other hand, it is known that KRR may perform dramatically worse than KPCR in these settings due to the saturation effect (see, e.g., Caponnetto and De Vito, 2007; Dicker, 2016; Dhillon et al., 2013).”