The jamming transition as a paradigm to understand the loss landscape of deep neural networks

By Mario Geiger, Stefano Spigler, Stéphane d'Ascoli, Levent Sagun, Marco Baity-Jesi, Giulio Biroli, Matthieu Wyart. 2018

157 citations recorded by the source corpus.

Deep learning has been immensely successful at a variety of tasks, ranging from classification to artificial intelligence. Learning corresponds to fitting training data, which is implemented by descending a very high-dimensional loss function. Understanding under which conditions neural networks do not get stuck in poor minima of the loss, and how the landscape of that loss evolves as depth is increased, remains a challenge. Here we predict, and test empirically, an analogy between this landscape and the energy landscape of repulsive ellipses. We argue that in fully connected deep networks a phase transition delimits the over- and underparametrized regimes where fitting can or cannot be achieved. In the vicinity of this transition, properties of the curvature of the minima of the loss (the spectrum of the Hessian) are critical. This transition shares direct similarities with the jamming transition by which particles form a disordered solid as the density is increased, which also occurs in certain classes of computational optimization and learning problems such as the perceptron. Our analysis gives a simple explanation as to why poor minima of the loss cannot be encountered in the overparametrized regime. Interestingly, we observe that the ability of fully connected networks to fit random data is independent of their depth, an independence that appears to also hold for real data. We also study a quantity Δ which characterizes how well (Δ<0) or badly (Δ>0) a datum is learned. At the critical point it is power-law distributed on several decades, P_{+}(Δ)∼Δ^{θ} for Δ>0 and P_{-}(Δ)∼(-Δ)^{-γ} for Δ<0, with exponents that depend on the choice of activation function. This observation suggests that near the transition the loss landscape has a hierarchical structure and that the learning dynamics is prone to avalanche-like dynamics, with abrupt changes in the set of patterns that are learned.

Subfields: Condensed Matter Physics, Materials Chemistry, Statistical and Nonlinear Physics, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Condensed Matter Physics

In Materials Chemistry

In Statistical and Nonlinear Physics

In Scaling & Learning Theory

Citations in the corpus

Deep double descent: where bigger models and more data hurt (2019, neutral): “…work provides asymptotic analysis for ridge regression over random features Similar double descent behavior was investigated in Opper (1995; 2001) Geiger et al. (2019b) showed that deep fully connected networks trained on the MNIST dataset with hinge loss exhibit a “jamming transition” when the…”
A jamming transition from under- to over-parametrization affects generalization in deep learning (2018, methodology): “In this section we recall in detail the analogy between jamming and supervised learning for deep neural networks [17, 18].”
Disentangling feature and lazy training in deep neural networks (2019, background): “Keep in mind that this criterion makes sense the overparametrized phase (as it will be the case in what follows), where networks manage to ﬁt all the training set (Spigler et al 2018).”
Triple descent and the two kinds of overfitting: where and why do they appear? (2020, unknown): “This peak1 was shown to be related (1)Also called the jamming peak due to similarities with a well-studied phenomenon in the Statistical Physics literature [14, 15, 16, 17, 18].”
Classification vs regression in overparameterized regimes: Does the loss function matter? (2020, background): “The phenomenon of overparameterization and interpolation yielding significantly improved empirical performance across a variety of models as well as tasks [32, 46, 18, 7] has received significant research attention over the last few years.”
Finding the Needle in the Haystack with Convolutions: on the benefits of architectural bias (2019, background): “…in this tradeoff argue that model size is not always a good predictor for overﬁtting [Neyshabur et al., 2018, Zhang et al., 2016, Neal et al., 2018, Geiger et al., 2019, Belkin et al., 2018], and consider instead other complexity measures of the function class, which favor CNNs due to their…”
The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks (2021, background): “Understanding the loss landscape of deep neural networks has been the subject of many studies due to its close connections to optimization and generalization (Li et al., 2017; Mei et al., 2018; Geiger et al., 2019; Nguyen et al., 2018; Fort et al., 2019; Baldassi et al., 2020).”
A jamming transition from under- to over-parametrization affects loss landscape and generalization (2018, background): “Understanding the connection with our findings (see also [18]) is certainly worth future studies.”
Heavy-Tailed Universality Predicts Trends in Test Accuracies for Very Large Pre-Trained Deep Neural Networks (2019, background): “Indeed, the dynamics of learning in DNNs seems to resemble a system near a phase transition, such as the phase boundary of spin glass, or a system displaying Self Organized Criticality (SOC), or a Jamming transition [25, 26].”
Disentangling feature and lazy learning in deep neural networks: an empirical study (2019, background): “…can be removed by ensemble-averaging output functions obtained with different initial conditions, leading to excellent performance already near the underparametrized to overparametrized (or jamming transition) h ∗ beyond which all the training data are correctly ﬁtted [Geiger et al., 2018].”