High-dimensional dynamics of generalization error in neural networks

By Madhu S. Advani, Andrew M. Saxe. 2017

540 citations recorded by the source corpus.

We perform an analysis of the average generalization dynamics of large neural networks trained using gradient descent. We study the practically-relevant “high-dimensional” regime where the number of free parameters in the network is on the order of or even larger than the number of examples in the dataset. Using random matrix theory and exact solutions in linear models, we derive the generalization error and training error dynamics of learning and analyze how they depend on the dimensionality of data and signal to noise ratio of the learning problem. We find that the dynamics of gradient descent learning naturally protect against overtraining and overfitting in large networks. Overtraining is worst at intermediate network sizes, when the effective number of free parameters equals the number of samples, and thus can be reduced by making a network smaller or larger. Additionally, in the high-dimensional regime, low generalization error requires starting with small initial weights. We then turn to non-linear neural networks, and show that making networks very large does not harm their generalization performance. On the contrary, it can in fact reduce overtraining, even without early stopping or regularization of any sort. We identify two novel phenomena underlying this behavior in overcomplete models: first, there is a frozen subspace of the weights in which no learning occurs under gradient descent; and second, the statistical properties of the high-dimensional regime yield better-conditioned input correlations which protect against overtraining. We demonstrate that standard application of theories such as Rademacher complexity are inaccurate in predicting the generalization performance of deep neural networks, and derive an alternative bound which incorporates the frozen subspace and conditioning effects and qualitatively matches the behavior observed in simulation.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

In Scaling & Learning Theory

Citations in the corpus

Scaling Laws for Neural Language Models (2020, background): “Overﬁtting should be related to the variance or the signal-to-noise ratio of the dataset [AS17], and this scales as 1 /D .”
Scaling description of generalization with number of parameters in deep learning (2019, background): “This phenomenon shares some similarity with the norm divergence that occurs in linear networks with mean square loss for which ||fN ||μ ∼ |N − P |−2 [27, 28].”
Implicit Regularization of Discrete Gradient Dynamics in Deep Linear Neural Networks (2019, positive): “, 2013, 2014, Advani and Saxe, 2017, Saxe et al., 2018, Lampinen and Ganguli, 2019] studied the gradient dynamics of two-layer linear networks and proved a result similar to our Thm. 2. We consider Saxe et al. [2018] as the closest related work, we re-use their notion of simple deep linear neural ne”
Reconciling modern machine-learning practice and the classical bias–variance trade-off (2018, neutral): “We note that this transition from under-to over-parameterized regimes for neural networks was also previously observed by [7, 1, 27, 37].”
The jamming transition as a paradigm to understand the loss landscape of deep neural networks (2018, background): “[72] Madhu S Advani and Andrew M Saxe, “Highdimensional dynamics of generalization error in neural networks,” arXiv preprint arXiv:1710.”
Scaling Laws for Neural Machine Translation (2021, background): “…at scale, theoretical understanding of the factors governing the quality and training dynamics of large neural networks has also being developing (Advani et al., 2020; Rosenfeld et al., 2019; Geiger et al., 2020; Ghorbani et al., 2021; Cho, 2020; Hutter, 2021; Bahri et al., 2021; Loureiro et…”
Continual Learning in the Teacher-Student Setup: Impact of Task Similarity (2021, background): “, 1993; Engel & Van den Broeck, 2001; Zdeborová & Krzakala, 2016) and has recently experienced a surge of activity in the machine learning community (Zimmer et al., 2014; Zhong et al., 2017; Tian, 2017; Du et al., 2018; Soltanolkotabi et al., 2018; Aubin et al., 2018; Saxe et al., 2018; Baity-Jesi e”
An Empirical Study of Example Forgetting during Deep Neural Network Learning (2018, background): “The study of the generalization properties of deep neural networks when trained by stochastic gradient descent has been the focus of several recent publications (Zhang et al., 2016; Keskar et al., 2016; Chaudhari et al., 2016; Advani & Saxe, 2017).”
Similarity of Neural Network Representations Revisited (2019, background): “Previous work (e.g. Advani & Saxe (2017); Amari et al. (2018); Saxe et al. (2014)) has made progress in understanding the theoretical dynamics of the neural network training process.”
A Study on Overfitting in Deep Reinforcement Learning (2018, positive): “Similar phenomenons are reported in supervised learning (Zhang et al., 2017), and recent work shows various approaches to explain this (Bartlett et al., 2017; Neyshabur et al., 2017; Advani & Saxe, 2017; Smith & Le, 2017; Dziugaite & Roy, 2017; Liang et al., 2017).”
A jamming transition from under- to over-parametrization affects generalization in deep learning (2018, background): “Among previous studies, [28] stands out as a natural predecessor of our work.”
An analytic theory of generalization dynamics and transfer learning in deep linear networks (2018, background): “We note that a related prior work (Advani & Saxe, 2017) studied generalization in shallow and deep linear networks, but that work was limited to networks with a single output, thereby precluding the possibility of addressing the issue of transfer learning.”
Two models of double descent for weak features (2019, background): “This risk curve reconciles the classical bias-variance trade-oﬀ with the behavior of predictive models that interpolate training data, as observed for several model families (including neural networks) in a wide variety of applications [BO98; AS17; Spi+18; Bel+18].”
Fix your classifier: the marginal value of training the last weight layer (2018, background): “…theoretical results related to neural network training (Soudry & Hoffer, 2017; Xie et al., 2016; Safran & Shamir, 2016; Soltanolkotabi et al., 2017; Soudry & Carmon, 2016) and even generalization (Gunasekar et al., 2017; Advani & Saxe, 2017; Wilson et al., 2017) usually assume over-speciﬁcation.”
On the geometry of generalization and memorization in deep neural networks (2021, result): “For tractability, we analyze gradient descent instead of SGD, consistent with other work (Lampinen and Ganguli, 2018; Advani and Saxe, 2017).”