A Modern Take on the Bias-Variance Tradeoff in Neural Networks

By Brady Neal, Sarthak Mittal, Aristide Baratin, Vinayak Tantia, Matthew Scicluna, Simon Lacoste-Julien, Ioannis Mitliagkas. 2018

186 citations recorded by the source corpus.

The bias-variance tradeoff tells us that as model complexity increases, bias falls and variances increases, leading to a U-shaped test error curve. However, recent empirical results with over-parameterized neural networks are marked by a striking absence of the classic U-shaped test error curve: test error keeps decreasing in wider networks. This suggests that there might not be a bias-variance tradeoff in neural networks with respect to network width, unlike was originally claimed by, e.g., Geman et al. (1992). Motivated by the shaky evidence used to support this claim in neural networks, we measure bias and variance in the modern setting. We find that both bias and variance can decrease as the number of parameters grows. To better understand this, we introduce a new decomposition of the variance to disentangle the effects of optimization and data sampling. We also provide theoretical analysis in a simplified setting that is consistent with our empirical findings.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

In Scaling & Learning Theory

Citations in the corpus

Scaling description of generalization with number of parameters in deep learning (2019, positive): “Remarkably, in our experiments, ensemble-averaging with n = 20 leads to a nearly flat test error for N > N∗; this supports the hypothesis that the improvement of generalization performance with N originates from reduced variance of fN when N gets large, as recently observed for mean-square regressio”
Reconciling modern machine-learning practice and the classical bias–variance trade-off (2018, neutral): “We note that this transition from under-to over-parameterized regimes for neural networks was also previously observed by [7, 1, 27, 37].”
Finding the Needle in the Haystack with Convolutions: on the benefits of architectural bias (2019, background): “…of optimization in this tradeoff argue that model size is not always a good predictor for overﬁtting [Neyshabur et al., 2018, Zhang et al., 2016, Neal et al., 2018, Geiger et al., 2019, Belkin et al., 2018], and consider instead other complexity measures of the function class, which favor CNNs…”
Implicit Regularization via Neural Feature Alignment (2020, background): “This seems at odds with the usual understanding of the bias-variance trade-oﬀ (Geman et al., 1992; Neal et al., 2018; Belkin et al., 2019): highly complex models are expected to overﬁt the training data and perform poorly on test data (Hastie et al., 2009).”
Disentangling feature and lazy learning in deep neural networks: an empirical study (2019, background): “This surprising behavior was related to ﬂuctuations δf of the output function induced by initial conditions, observed to decrease with h [Neal et al., 2019, Geiger et al., 2019].”
A statistical perspective on distillation (2021, background): “In the context of neural networks, recent works have challenged the conventional wisdom (Geman et al., 1992) of increased model complexity (e.g., increased network width and/or depth) implying a higher variance (Neal et al., 2018; Yang et al., 2020).”
Disentangling feature and lazy training in deep neural networks (2019, background): “A surprising observation is that their performance on supervised tasks keeps increasing with their width h in the over-parametrized regime where they already ﬁt all the training data (Neyshabur et al 2017, Bansal et al 2018, Advani and Saxe 2017, Spigler et al 2018).”
A Survey of End-to-End Driving: Architectures and Training Methods (2020, background): “Training instability: Another challenge of using neural networks as driving models is that the training process is unstable and is not guaranteed to converge to the same minima each time [57].”
Triple descent and the two kinds of overfitting: where and why do they appear? (2020, background): “Instead, recent developments show that deep neural networks, as well as other machine learning models, exhibit a starkly different behaviour [8, 9, 10, 11, 12].”