High-Dimensional Asymptotics of Prediction: Ridge Regression and Classification

By Edgar Dobriban, Stefan Wager. 2015

342 citations recorded by the source corpus.

We provide a unified analysis of the predictive risk of ridge regression and regularized discriminant analysis in a dense random effects model. We work in a high-dimensional asymptotic regime where $p, n \to \infty$ and $p/n \to \gamma \in (0, \, \infty)$, and allow for arbitrary covariance among the features. For both methods, we provide an explicit and efficiently computable expression for the limiting predictive risk, which depends only on the spectrum of the feature-covariance matrix, the signal strength, and the aspect ratio $\gamma$. Especially in the case of regularized discriminant analysis, we find that predictive accuracy has a nuanced dependence on the eigenvalue distribution of the covariance matrix, suggesting that analyses based on the operator norm of the covariance matrix may not be sharp. Our results also uncover several qualitative insights about both methods: for example, with ridge regression, there is an exact inverse relation between the limiting predictive risk and the limiting estimation risk given a fixed signal strength. Our analysis builds on recent advances in random matrix theory.

Subfields: Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/e26aa4b56e43a6d1360413f205aa79aa2386fda3

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Statistics and Probability

Citations in the corpus

Exact minimax risk for linear least squares, and the lower tail of sample covariance matrices (2019, background): “Another line of work [EK13, Dic16, DM16, EK18, DW18] considers the limiting behavior of regression procedures in the high-dimensional asymptotic regime where d, n tend to inﬁnity at a proportional rate, with their ratio kept constant [Hub73].”
High-dimensional regression adjustments in randomized experiments (2016, methodology): “Following [34], suppose moreover that the true parameters β and β((1)) are independently and randomly drawn from a random effects model with”
Capturing the learning curves of generic features maps for realistic data sets with a teacher-student model (2021, methodology): “For ridge regression, there are also precise predictions thanks to random matrix theory [32, 12, 33, 34, 35].”
Surprises in High-Dimensional Ridgeless Least Squares Interpolation (2019, neutral): “Very similar (though not identical) results can be found in Dicker [22], Dobriban and Wager [23].”
The Optimal Ridge Penalty for Real-world High-dimensional Data Can Be Zero or Negative due to the Implicit Ridge Regularization (2020, background): “This result might appear to contradict the literature; for example, Dobriban and Wager (2018) and later Hastie et al. (2019) studied high-dimensional asymptotics of ridge regression performance for p, n → ∞ while p/n = γ and proved, among other things, that the optimal λ is always positive.”
Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisited (2020, methodology): “We demonstrate this property algebraically in Appendix B, where we also connect the effective dimensionality to the bias-variance tradeoff (Dobriban & Wager, 2018) and to the Hilbert space norm (Rasmussen & Williams, 2008).”