The Generalization Error of Random Features Regression: Precise Asymptotics and the Double Descent Curve

By Song Mei, Andrea Montanari. 2019

722 citations recorded by the source corpus.

Deep learning methods operate in regimes that defy the traditional statistical mindset. Neural network architectures often contain more parameters than training samples, and are so rich that they can interpolate the observed labels, even if the latter are replaced by pure noise. Despite their huge complexity, the same architectures achieve small generalization error on real data.

Subfields: Artificial Intelligence, Statistics and Probability, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

In Scaling & Learning Theory

Citations in the corpus

Scaling description of generalization with number of parameters in deep learning (2019, methodology): “Very recently in [38], a rigorous derivation of the double descent curve was obtained for the mean square regression of simple functions using random features models.”
Underspecification Presents Challenges for Credibility in Modern Machine Learning (2020, methodology): “Building on the work of Mei and Montanari (2019) we can determine the precise asymptotics of the risk under certain distribution shifts in the limit n,N, d→∞ with fixed ratios n/d, N/n.”
Deep double descent: where bigger models and more data hurt (2019, positive): “Similar behavior was previously observed in Opper (1995; 2001), Advani & Saxe (2017), Spigler et al. (2018), and Geiger et al.”
An Investigation of Why Overparameterization Exacerbates Spurious Correlations (2020, methodology): “We used a modified ResNet10 with variable widths, following the approach in Nakkiran et al. (2019) and extending the torchvision implementation.”
Artemis: tight convergence guarantees for bidirectional compression in Federated Learning (2020, background): “For instance, Seide et al. [2014] are one of the first to propose to quantize each gradient component by either −1 or 1.”
Graph Neural Networks Exponentially Lose Expressive Power for Node Classification (2019, methodology): “For example, while Belkin et al. (2019) employed the interpolating hypothesis with the minimum norm, Mei & Montanari (2019) found that the double descent was alleviated or disappeared when they used Ridge-type regularization techniques.”
Explaining neural scaling laws (2021, background): “This concentration result with respect to dataset size has appeared for linear models in [27] and for single hidden layer networks with high-dimensional input data in [32–34].”
Precise Tradeoffs in Adversarial Training for Linear Regression (2020, background): “(2018); Mei and Montanari (2019). For least square regression with Gaussian covariates, it is shown that the global minimum of the risk is achieved in the underparametrized setting δ > 1 (unless miss-specified structures are assumed).”
Capturing the learning curves of generic features maps for realistic data sets with a teacher-student model (2021, methodology): “The asymptotic behavior of kernel matrices has indeed been the subject of intense scrutiny [45, 62, 46, 11, 63, 48].”
Think Locally, Act Globally: Federated Learning with Local and Global Representations (2020, background): “While simple in setup, its behavior is rich and has proved insightful in the understanding of nonlinear models as well [40, 39, 21].”
Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural networks (2020, background): “This non-monotonic behavior is reminiscent of the recently described “double-descent” phenomenon [3, 33, 34, 5], where generalization error is non-monotonic as a function of model complexity in many modern machine learning models.”
Surprises in High-Dimensional Ridgeless Least Squares Interpolation (2019, methodology): “After a first appearance of this manuscript, universality has been investigated in the context of neural networks in several papers [47, 50, 29, 31, 37, 1].”
The Optimal Ridge Penalty for Real-world High-dimensional Data Can Be Zero or Negative due to the Implicit Ridge Regularization (2020, background): “…al., 2019; Belkin et al., 2018a,b, 2019a,b,c; Nakkiran, 2019; Nakkiran et al., 2020a,b; Liang and Rakhlin, 2020; Hastie et al., 2019; Bartlett et al., 2020; Chinot and Lerasle, 2020; Muthukumar et al., 2020; Mei and Montanari, 2019; Bibas et al., 2019; Dereziński et al., 2019; Negrea et al., 2019).”
Triple descent and the two kinds of overfitting: where and why do they appear? (2020, positive): “On the analytical side, our work directly uses the results for high-dimensional random features models derived in [11, 37] (for the test loss), [38] (for the spectral analysis) and [20] (for the bias-variance decomposition).”
Modelling the influence of data structure on learning in neural networks (2019, background): “…played a major role in theoretical studies of the generalisation ability of neural networks from an average-case perspective, particularly within the framework of statistical mechanics [3, 12, 4, 5, 13, 14, 15, 16, 17], and also in recent statistical learning theory works, e.g. [18, 19, 20, 21].”