A jamming transition from under- to over-parametrization affects loss landscape and generalization

By Stefano Spigler, Mario Geiger, Stéphane d'Ascoli, Levent Sagun, Giulio Biroli, Matthieu Wyart. 2018

59 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Scaling & Learning Theory

Citations in the corpus

Scaling description of generalization with number of parameters in deep learning (2019, methodology): “When optimizing neural networks using the hinge loss, there is a sharp phase transition (similar to the jamming transition that occurs in granular materials [21]) at some N∗(P ) such that for N ≥ N∗ the dynamic process reaches global minima of the loss [18, 40].”
Large Memory Layers with Product Keys (2019, background): “The higher-capacity enables better modeling of data like natural text or images, and it also improves generalization [40, 31].”
Reconciling modern machine-learning practice and the classical bias–variance trade-off (2018, background): “In particular, [37] draws a connection to the physical phenomenon of “jamming” in particle systems.”
Heavy-Tailed Universality Predicts Trends in Test Accuracies for Very Large Pre-Trained Deep Neural Networks (2019, background): “Indeed, the dynamics of learning in DNNs seems to resemble a system near a phase transition, such as the phase boundary of spin glass, or a system displaying Self Organized Criticality (SOC), or a Jamming transition [25, 26].”
Finite Depth and Width Corrections to the Neural Tangent Kernel (2019, result): “Assuming this trajectory turns out to be data-dependent, our results suggest that the double descent curve [1, 2, 21] that trades oﬀ complexity vs. generalization error may display signiﬁcantly diﬀerent behaviors depending on the mode of network overparameterization.”
Surprises in High-Dimensional Ridgeless Least Squares Interpolation (2019, background): “[61] Stefano Spigler, Mario Geiger, Stephane d’Ascoli, Levent Sagun, Giulio Biroli, and Matthieu Wyart.”
Generalization of Two-layer Neural Networks: An Asymptotic Viewpoint (2020, background): “Recently, a series of works highlighted the implicit regularization of interpolators in the overparameterized regime (Belkin et al., 2018; Spigler et al., 2018; Geiger et al., 2018; Advani and Saxe, 2017).”
A Modern Take on the Bias-Variance Tradeoff in Neural Networks (2018, background): “However, there is a growing amount of empirical evidence that wider networks generalize better than their smaller counterparts (Neyshabur et al., 2015; Zagoruyko & Komodakis, 2016; Novak et al., 2018; Lee et al., 2018; Belkin et al., 2018; Spigler et al., 2018; Liang et al., 2017; Canziani et al., 2”
Disentangling feature and lazy training in deep neural networks (2019, background): “Keep in mind that this criterion makes sense the overparametrized phase (as it will be the case in what follows), where networks manage to ﬁt all the training set (Spigler et al 2018).”
Disentangling feature and lazy learning in deep neural networks: an empirical study (2019, methodology): “…terms, our observations suggest that the best procedure consists in ensemble averaging networks in the feature-learning regime, with h that can be small yet sufﬁciently large to ﬁt the data (i.e. past the jamming transition h ∗ , below which performance is known to degrade [Spigler et al., 2018]).”