A jamming transition from under- to over-parametrization affects generalization in deep learning

By Stefano Spigler, Mario Geiger, Stéphane d'Ascoli, Levent Sagun, Giulio Biroli, Matthieu Wyart. 2018

160 citations recorded by the source corpus.

In this paper we first recall the recent result that in deep networks a phase transition, analogous to the jamming transition of granular media, delimits the over- and under-parametrized regimes where fitting can or cannot be achieved. The analysis leading to this result support that for proper initialization and architectures, in the whole over-parametrized regime poor minima of the loss are not encountered during training, because the number of constraints that hinders the dynamics is insufficient to allow for the emergence of stable minima. Next, we study systematically how this transition affects generalization properties of the network (i.e. its predictive power). As we increase the number of parameters of a given model, starting from an under-parametrized network, we observe for gradient descent that the generalization error displays three phases: (i) initial decay, (ii) increase until the transition point—where it displays a cusp—and (iii) slow decay toward an asymptote as the network width diverges. However if early stopping is used, the cusp signaling the jamming transition disappears. Thereby we identify the region where the classical phenomenon of over-fitting takes place as the vicinity of the jamming transition, and the region where the model keeps improving with increasing the number of parameters, thus organizing previous empirical observations made in modern neural networks.

Subfields: Artificial Intelligence, Atomic and Molecular Physics, and Optics, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Atomic and Molecular Physics, and Optics

In Scaling & Learning Theory

Citations in the corpus

Scaling description of generalization with number of parameters in deep learning (2019, background): “Numerical and theoretical studies [6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17] show that in the overparametrized regime, the loss landscape of DNNs is not rough with isolated minima as initially thought [18, 19], but instead has connected level sets and presents many flat directions, even near its g”
High-dimensional dynamics of generalization error in neural networks (2017, positive): “12A), consistent with recent results on the error landscape of ReLU networks (Soudry & Hoffer, 2017; Spigler et al., 2018).”
Deep double descent: where bigger models and more data hurt (2019, positive): “Similar behavior had been observed in Opper (1995; 2001), Advani & Saxe (2017), Spigler et al. (2018), and Geiger et al. (2019b).”
Reconciling modern machine-learning practice and the classical bias–variance trade-off (2018, background): “In particular, [37] draws a connection to the physical phenomenon of “jamming” in particle systems.”
The jamming transition as a paradigm to understand the loss landscape of deep neural networks (2018, background): “[46] Stefano Spigler, Mario Geiger, Stéphane d’Ascoli, Levent Sagun, Giulio Biroli, and Matthieu Wyart, “A jamming transition from under-to over-parametrization affects loss landscape and generalization,” NIPS Workshop “Integration of Deep Learning Theories” contribution, arXiv preprint arXiv:1810.”
Triple descent and the two kinds of overfitting: where and why do they appear? (2020, unknown): “In the absence of regularization, increasing P and N respectively yields parameter-wise and sample-wise double descent curves [8, 9, 10, 11, 12, 13], whereby the generalization error first decreases, then peaks at the interpolation threshold (at which point training error vanishes), then decreases m”
Geometric compression of invariant manifolds in neural nets (2020, background): “Throughout this work, it is assumed that the network width h is sufficiently large for the algorithm to operate in the overparametrized regime [17, 18].”
The Optimal Ridge Penalty for Real-world High-dimensional Data Can Be Zero or Negative due to the Implicit Ridge Regularization (2020, background): “…but is in line with the large body of ongoing research kindled by Zhang et al. (2017) and mostly done in parallel to our work (Advani et al., 2020; Spigler et al., 2019; Belkin et al., 2018a,b, 2019a,b,c; Nakkiran, 2019; Nakkiran et al., 2020a,b; Liang and Rakhlin, 2020; Hastie et al., 2019;…”
Loss landscapes and optimization in over-parameterized non-linear systems and neural networks (2020, background): “Statistically, over-parameterized systems enter a new interpolating regime, where increasing the number of parameters, even indeﬁnitely to inﬁnity, can improve generalization [5, 34].”
Finite Depth and Width Corrections to the Neural Tangent Kernel (2019, result): “Assuming this trajectory turns out to be datadependent, our results suggest that the double descent curve [1, 2, 21] that trades off complexity vs.”
Surprises in High-Dimensional Ridgeless Least Squares Interpolation (2019, background): “[63], who studied the “jamming transition” from underparametrized to overparametrized neural networks.”
Locality defeats the curse of dimensionality in convolutional teacher–student scenarios (2021, background): “The α’s control the smoothness of these functions, in the sense that, if α>n ∈ N, then the constituent kernel C is at least n times differentiable [34].”
Generalization of Two-layer Neural Networks: An Asymptotic Viewpoint (2020, background): “Recently, a series of works highlighted the implicit regularization of interpolators in the overparameterized regime (Belkin et al., 2018; Spigler et al., 2018; Geiger et al., 2018; Advani and Saxe, 2017).”
Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural networks (2020, background): “…machine learning models, signaled by a peak and a subsequent descent in generalization error as the model complexity, or the number of parameters, increases, and the model transitions from an under-parameterized to over-parameterized (interpolating) regime 3,5,35,38,42,43,54,66,74,75 .”