Reconciling modern machine learning and the bias-variance trade-off

By Mikhail Belkin, Daniel J. Hsu, Siyuan Ma, Soumik Mandal. 2018

136 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Evaluation & Benchmarks

Citations in the corpus

Scaling Laws for Neural Language Models (2020, background): “Various works [AS17, BHMM18] have investigated generalization in highly overparameterized models, ﬁnding a “jamming transition” [GJS + 19] when the model size reaches the dataset size (this may require training many orders of magnitude beyond typical practice, and in particular does not use early…”
Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers (2020, background): “For instance, it studies forms of implicit regularization (Zhang et al., 2017; Belkin et al., 2018), characterizes the expressivity of deep models (Raghu et al.”
Underspecification Presents Challenges for Credibility in Modern Machine Learning (2020, background): “Various notions of underspecification are well-documented in the ML literature; for example, it is a core idea in deep ensembles, double descent, Bayesian deep learning, and loss landscape analysis (Lakshminarayanan et al., 2017; Fort et al., 2019; Belkin et al., 2018; Nakkiran et al., 2020).”
Deep double descent: where bigger models and more data hurt (2019, positive): “This phenomenon was first postulated in generality by Belkin et al. (2018) who named it “double descent”, and demonstrated it for decision trees, random features, and 2-layer neural networks with `2 loss, on a variety of learning tasks including MNIST and CIFAR-10.”
Generalization Guarantees for Neural Networks via Harnessing the Low-rank Structure of the Jacobian (2019, background): “An interesting line of work establish connection between kernel methods and neural networks and study the generalization abilities of kernel methods when the model interpolates the training data [19, 10, 11, 12, 38, 13].”
Behaviour Suite for Reinforcement Learning (2019, background): “It is only now, several years after these high proﬁle successes, that learning theory has begun to turn its attention back to deep learning (Kawaguchi, 2016; Bartlett et al., 2017; Belkin et al., 2018).”
Does learning require memorization? a short tale about a long tail (2019, background): “Recent work of Belkin et al. [BHMM18] shows empirically and also on simple synthetic models [BHX19] that classical bias-variance trade-off curve may coexist with the interpolating regime in which the generalization error improves as the class capacity grows.”
A jamming transition from under- to over-parametrization affects generalization in deep learning (2018, background): “The described cusp in the generalization error has been observed empirically in [29] for random forest models and simple neural networks.”
Benign overfitting in linear regression (2019, background): “Belkin, Hsu, Ma and Mandal [12] studied experimentally the excess risk as a function of the dimension of a sequence of parameter spaces for linear and non-linear classes.”
Finding the Needle in the Haystack with Convolutions: on the benefits of architectural bias (2019, background): “…that model size is not always a good predictor for overﬁtting [Neyshabur et al., 2018, Zhang et al., 2016, Neal et al., 2018, Geiger et al., 2019, Belkin et al., 2018], and consider instead other complexity measures of the function class, which favor CNNs due to their smaller complexity Du et…”
Toward Moderate Overparameterization: Global Convergence Guarantees for Training Shallow Neural Networks (2019, background): “This is the subject of a few interesting recent papers [5, 36, 41–47].”
Harmless interpolation of noisy data in regression (2019, background): “These phenomena are being actively investigated in a statistical sense [5,6] and a computational sense [7–9] in classiﬁcation problems and/or noiseless models.”
A Modern Take on the Bias-Variance Tradeoff in Neural Networks (2018, background): “(2018); Belkin et al. (2018) point out that generalization error acts according to conventional wisdom in the under-parameterized setting, that it decreases with capacity in the over-parameterized setting, and that there is a sharp transition between the two settings.”
On the Risk of Minimum-Norm Interpolants and Restricted Lower Isometry of Kernels (2019, background): “…work on nonparametric local rules (Belkin et al., 2018b,d), high-dimensional linear regression (Bartlett et al., 2019; Hastie et al., 2019), random features model (Ghorbani et al., 2019), and kernel (ridgeless) regression (Belkin et al., 2018c; Liang and Rakhlin, 2018; Rakhlin and Zhai, 2018).”
Triple descent and the two kinds of overfitting: where and why do they appear? (2020, unknown): “In the absence of regularization, increasing P and N respectively yields parameter-wise and sample-wise double descent curves [8, 9, 10, 11, 12, 13], whereby the generalization error first decreases, then peaks at the interpolation threshold (at which point training error vanishes), then decreases m”
Understanding overfitting peaks in generalization error: Analytical risk curves for l2 and l1 penalized interpolation (2019, background): “An interesting related phenomenon has been noted: the existence of a peak in GE with increasing fitting model complexity[2, 1, 10, 11].”