Asymptotic behavior of unregularized and ridge-regularized high-dimensional robust regression estimators : rigorous results

By Noureddine El Karoui. 2013

113 citations recorded by the source corpus.

We study the behavior of high-dimensional robust regression estimators in the asymptotic regime where $p/n$ tends to a finite non-zero limit. More specifically, we study ridge-regularized estimators, i.e $\widehat{\beta}=\text{argmin}_{\beta \in \mathbb{R}^p} \frac{1}{n}\sum_{i=1}^n \rho(\varepsilon_i-X_i' \beta)+\frac{\tau}{2}\lVert\beta\rVert^2$. In a recently published paper, we had developed with collaborators probabilistic heuristics to understand the asymptotic behavior of $\widehat{\beta}$. We give here a rigorous proof, properly justifying all the arguments we had given in that paper. Our proof is based on the probabilistic heuristics we had developed, and hence ideas from random matrix theory, measure concentration and convex analysis. While most the work is done for $\tau>0$, we show that under some extra assumptions on $\rho$, it is possible to recover the case $\tau=0$ as a limiting case. We require that the $X_i$'s be i.i.d with independent entries, but our proof handles the case where these entries are not Gaussian. A 2-week old paper of Donoho and Montanari [arXiv:1310.7320] studied a similar problem by a different method and with a different point of view. At this point, their interesting approach requires Gaussianity of the design matrix.

Subfields: Computational Mechanics, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/a11c0f52307631289ee07a5b6e6079b994dda372

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Statistics and Probability

Citations in the corpus

Exact minimax risk for linear least squares, and the lower tail of sample covariance matrices (2019, background): “Another line of work [25, 21, 23, 26, 22] considers the limiting behavior of regression procedures in the high-dimensional asymptotic regime where d, n tend to infinity at a proportional rate, with their ratio kept constant [35].”
A modern maximum-likelihood theory for high-dimensional logistic regression (2018, background): “Later on, the regime where p is comparable to n became the subject of a series of remarkable works [5,22,26–28]; these works only concern the linear model.”
High-Dimensional Asymptotics of Prediction: Ridge Regression and Classification (2015, background): “Important references include, among others, Bickel and Levina (2004), Dicker (2014), El Karoui (2013), Fujikoshi et al. (2011), Hsu et al. (2014), Saranadasa (1993), and Zollanvari and Dougherty (2015).”
A Random Matrix Approach to Neural Networks (2017, background): “Notable exceptions are the line of works surrounding kernel random matrices (El Karoui, 2010; Couillet and Benaych-Georges, 2016) as well as large dimensional robust statistics models (Couillet, Pascal and Silverstein, 2015; El Karoui, 2013; Zhang, Cheng and Singer, 2014).”
Statistical Mechanics of Optimal Convex Inference in High Dimensions (2016, unknown): “[30,31,38], and they illustrate the severity of”
On the impact of predictor geometry on the performance on high-dimensional ridge-regularized generalized robust regression estimators (2018, methodology): “See also the paper [9] that handles only the case of i.i.d Gaussian predictors, whereas El Karoui [15] can deal with more general assumptions on the predictors.”