Deep double descent: where bigger models and more data hurt
By Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, Ilya Sutskever. 2019
1,162 citations recorded by the source corpus.
We show that a variety of modern deep learning tasks exhibit a ‘double-descent’ phenomenon where, as we increase model size, performance first gets worse and then gets better. Moreover, we show that double descent occurs not just as a function of model size, but also as a function of the number of training epochs. We unify the above phenomena by defining a new complexity measure we call the effective model complexity and conjecture a generalized double descent with respect to this measure. Furthermore, our notion of model complexity allows us to identify certain regimes where increasing (even quadrupling) the number of train samples actually hurts test performance.
Subfields: Artificial Intelligence, Scaling & Learning Theory
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Percy Liang
- Luke S. Zettlemoyer
- Yoshua Bengio
- Quoc V. Le
- Furu Wei
- Ruslan Salakhutdinov
In Scaling & Learning Theory
- Andrew Gordon Wilson
- Hanie Sedghi
- Pavel Izmailov
- Shotaro Akaho
- Ryo Karakida
- Neil Houlsby
- Matthew D. Hoffman
- Behnam Neyshabur
Citations in the corpus
- Scaling Laws for Transfer (2021, neutral): “The Z-shaped curves are a bit surprising, and are somewhat similar in shape to what was observed in [NKB + 19].”
- Underspecification Presents Challenges for Credibility in Modern Machine Learning (2020, background): “Various notions of underspecification are well-documented in the ML literature; for example, it is a core idea in deep ensembles, double descent, Bayesian deep learning, and loss landscape analysis (Lakshminarayanan et al., 2017; Fort et al., 2019; Belkin et al., 2018; Nakkiran et al., 2020).”
- Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets (2022, background): “A double descent of validation loss has been documented in some circumstances, but is considered unusual among practitioners Nakkiran et al. (2019); Belkin et al. (2018); d’Ascoli et al. (2020).”
- SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Network (2021, neutral): “It has been observed that these large models tend to enhance generalization [27].”
- Learning curves for continual learning in neural networks: Self-knowledge transfer and forgetting (2021, background): “For single-task training, however, many empirical and theoretical studies have succeeded in characterizing generalization performance in over-parameterized neural networks and given quantitative evaluation on sample-size dependencies, e.g., double descent (Nakkiran et al., 2020).”
- BERT-of-Theseus: Compressing BERT by Progressive Module Replacing (2020, background): “These models benefit from their “overparameterized” nature [24] and contain millions or even billions of parameters, making it computationally expensive and inefficient considering both memory consumption and high latency.”
- Bayesian Deep Learning and a Probabilistic Perspective of Generalization (2020, background): “[33], measuring both error and negative log likelihood (NLL).”
- The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks (2021, background): “This effect is reminiscent of the double descent phenomena (Belkin et al., 2019; Nakkiran et al., 2019).”
- The Optimal Ridge Penalty for Real-world High-dimensional Data Can Be Zero or Negative due to the Implicit Ridge Regularization (2020, methodology): “Moreover, increasing network size p can improve test-set performance even after p is large enough to ensure zero training error (Neyshabur, 2017; Nakkiran et al., 2020a), which is qualitatively similar to what we observed here.”
- Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisited (2020, methodology): “Nakkiran et al. (2019) found generalization gains as neural networks become highly overparameterized, showing the double descent phenomenon that occurs as the width parameter of both residual and convolutional neural networks is increased.”
- The Computational Limits of Deep Learning (2020, background): “Figure 1 (a) generalizes this, showing the increase in computation needed as the effective model complexity (d/s) increases [84].”
- Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural networks (2020, background): “These networks work in an overparameterized regime [3, 5] with much more parameters than training samples, and are highly expressive to a level that they can even fit random noise [2].”
- Model complexity of deep learning: a survey (2021, methodology): “Several previous studies [44, 60, 74, 81, 89] define and explore effective complexity measures mainly from the perspective of piecewise linear property [44, 81, 89], Fisher-Rao metric [60], or the size of trainable samples [74].”
- The Curious Case of Adversarially Robust Models: More Data Can Help, Double Descend, or Hurt Generalization (2020, background): “Empirically, Nakkiran et al. [2019] also discover a sample-wise double descent phenomenon.”