Scaling description of generalization with number of parameters in deep learning

By Mario Geiger, Arthur Jacot, Stefano Spigler, Franck Gabriel, Levent Sagun, Stéphane d'Ascoli, Giulio Biroli, Clément Hongler, Matthieu Wyart. 2019

214 citations recorded by the source corpus.

Supervised deep learning involves the training of neural networks with a large number N of parameters. For large enough N, in the so-called over-parametrized regime, one can essentially fit the training data points. Sparsity-based arguments would suggest that the generalization error increases as N grows past a certain threshold N*. Instead, empirical studies have shown that in the over-parametrized regime, generalization error keeps decreasing with N. We resolve this paradox through a new framework. We rely on the so-called Neural Tangent Kernel, which connects large neural nets to kernel methods, to show that the initialization causes finite-size random fluctuations of the neural net output function f N around its expectation . These affect the generalization error for classification: under natural assumptions, it decays to a plateau value in a power-law fashion ∼N−1/2. This description breaks down at a so-called jamming transition N = N*. At this threshold, we argue that diverges. This result leads to a plausible explanation for the cusp in test error known to occur at N*. Our results are confirmed by extensive empirical observations on the MNIST and CIFAR image datasets. Our analysis finally suggests that, given a computational envelope, the smallest generalization error is obtained using several networks of intermediate sizes, just beyond N*, and averaging their outputs.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Scaling & Learning Theory

Citations in the corpus

Scaling Laws for Neural Language Models (2020, methodology): “We train language models on WebText2, an extended version of the WebText [RWC + 19] dataset, tokenized using byte-pair encoding [SHB15] with a vocabulary size n vocab = 50257 .”
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding (2020, methodology): “This general tendency motivated recent studies to scrutinize the factors playing a critical role in the success of scaling [17, 18, 19, 20, 3], including the amounts of training data, the model size, and the computation being utilized as found by past studies.”
Deep double descent: where bigger models and more data hurt (2019, methodology): “Moreover, Geiger et al. (2019a) provide preliminary results for model-wise double descent in convolutional networks trained on CIFAR-10.”
The jamming transition as a paradigm to understand the loss landscape of deep neural networks (2018, background): “This has been observed, for instance, in [46, 76], and studied by the authors in [47] (preprint).”
Scaling Laws for Neural Machine Translation (2021, background): “…theoretical understanding of the factors governing the quality and training dynamics of large neural networks has also being developing (Advani et al., 2020; Rosenfeld et al., 2019; Geiger et al., 2020; Ghorbani et al., 2021; Cho, 2020; Hutter, 2021; Bahri et al., 2021; Loureiro et al., 2021).”
Orthogonal representations for robust context-dependent task performance in brains and neural networks (2022, positive): “In the rich regime, which occurs under low initial variance, the hidden units instead learn highly structured representations that are tailored to the task demands (Saxe et al., 2019; Geiger et al., 2020; Woodworth et al., 2020; Paccolat et al., 2021).”
A jamming transition from under- to over-parametrization affects generalization in deep learning (2018, methodology): “We have provided a quantitative explanation for this phenomenon in [33].”
A Modern Take on the Bias-Variance Tradeoff in Neural Networks (2018, background): “All approaches focus on test error, rather than studying bias and variance directly (Neyshabur et al., 2019; Geiger et al., 2019; Liang et al., 2017; Belkin et al., 2019a).”
Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit (2019, background): “This PDE has smoother solutions ρt that are supported everywhere and converge globally to a unique fixed point Mei et al. (2018). In this setting, we prove a dimension-free approximation guarantee for the case of bounded activations.”
MetaInit: Initializing learning by learning to initialize (2019, background): “We argue that low gradient quotient correlates with a number of recently identiﬁed predictors of trainability including the conditioning of the hessian [18], the Fisher Information [3], and of the neural tangent kernel [28, 17, 34].”
Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsing (2022, background): “The scaling behavior of many tasks have been shown to be predictable and generalization error generally decreases with scale (Geiger et al., 2019; Rosenfeld et al., 2020; Henighan et al., 2020), with some exceptions showing the limits of large scale pre-training (Abnar et al., 2021).”
Finding the Needle in the Haystack with Convolutions: on the benefits of architectural bias (2019, background): “On the one hand, several recent works studying the role of optimization in this tradeoff argue that model size is not always a good predictor for overfitting [29, 37, 28, 17, 6], and consider instead other complexity measures of the function class, which favor CNNs due to their smaller complexity [1”
A type of generalization error induced by initialization in deep neural networks (2019, background): “…of the training, that difference shrinks to hini− hK (·;0,X, hini(X)), which is the residual of fitting hini sampled at X by the same DNN. Note that Geiger et al. (2019)figures out qualitatively that hini, which does not vanish as the width of DNN tends to infinity, decreases during the training.”
Disentangling feature and lazy learning in deep neural networks: an empirical study (2019, background): “This surprising behavior was related to ﬂuctuations δf of the output function induced by initial conditions, observed to decrease with h [Neal et al., 2019, Geiger et al., 2019].”
Explaining neural scaling laws (2021, background): “If we further assume, however, that f T on average has been trained such as to be suﬃciently close to a local minimum of the loss, such that √ , then typical instances will also obey the O ( (cid:15) ) scaling.”
Asymptotics of Wide Networks from Feynman Diagrams (2019, background): “For additional works discussing the training dynamics of wide networks see [23, 24].”