Comparing dynamics: deep neural networks versus glassy systems

By Marco Baity-Jesi, Levent Sagun, Mario Geiger, Stefano Spigler, Gérard Ben Arous, Chiara Cammarota, Yann LeCun, Matthieu Wyart, Giulio Biroli. 2018

131 citations recorded by the source corpus.

We analyze numerically the training dynamics of deep neural networks (DNN) by using methods developed in statistical physics of glassy systems. The two main issues we address are (1) the complexity of the loss landscape and of the dynamics within it, and (2) to what extent DNNs share similarities with glassy systems. Our findings, obtained for different architectures and datasets, suggest that during the training process the dynamics slows down because of an increasingly large number of flat directions. At large times, when the loss is approaching zero, the system diffuses at the bottom of the landscape. Despite some similarities with the dynamics of mean-field glassy systems, in particular, the absence of barrier crossing, we find distinctive dynamical behaviors in the two cases, showing that the statistical properties of the corresponding loss and energy landscapes are different. In contrast, when the network is under-parametrized we observe a typical glassy behavior, thus suggesting the existence of different phases depending on whether the network is under-parametrized or over-parametrized.

Subfields: Atomic and Molecular Physics, and Optics, Materials Chemistry, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Atomic and Molecular Physics, and Optics

In Materials Chemistry

In Scaling & Learning Theory

Citations in the corpus

Scaling description of generalization with number of parameters in deep learning (2019, background): “Numerical and theoretical studies [6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17] show that in the overparametrized regime, the loss landscape of DNNs is not rough with isolated minima as initially thought [18, 19], but instead has connected level sets and presents many flat directions, even near its g”
A jamming transition from under- to over-parametrization affects generalization in deep learning (2018, background): “Furthermore, studies of the Hessian of the loss function [12, 13, 14] and of the learning dynamics [15, 16] support that the landscape is characterized by an abundance of flat directions, even near its bottom, at odds with traditional glasses.”
A high-bias, low-variance introduction to Machine Learning for physicists (2018, unknown): “At the same time, methods from statistical physics have been applied to the field of deep learning to study the thermodynamic efficiency of learning rules (Goldt and Seifert, 2017), to explore the hypothesis space that DNNs span, make analogies between training DNNs and spin glasses (Baity-Jesi et a”
The committee machine: computational to statistical gaps in learning a two-layers neural network (2018, background): “Similar large computational gaps have been previously identified in many problems and we believe that its identification in a multi-layer neural network model makes it a very interesting candidate for further mathematical studies of the energy landscape in deep learning [12, 47].”
Neural Networks as Interacting Particle Systems: Asymptotic Convexity of the Loss Landscape and Universal Scaling of the Approximation Error (2018, background): “Several recent papers have focused on the analysis of the shape and properties of the objective or “loss” function landscape [3,9,24].”
Generalization in multitask deep neural classifiers: a statistical physics approach (2019, background): “There has been a noticeable uptick in recent efforts to build a rigorous theoretical foundation for deep learning (see, e.g. [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] for a sampling of this trend).”
Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning (2018, background): “See [13, 72, 64, 10]—and recall that, while this connection is sometimes not explicitly noted, glasses are defined in terms of their slow dynamics.”
Disentangling feature and lazy training in deep neural networks (2019, background): “We ﬁnally investigate the temporal evolution of learning, known to be characterized by several time scales (Baity-Jesi et al 2018).”
Disentangling feature and lazy learning in deep neural networks: an empirical study (2019, methodology): “…terms, our observations suggest that the best procedure consists in ensemble averaging networks in the feature-learning regime, with h that can be small yet sufﬁciently large to ﬁt the data (i.e. past the jamming transition h ∗ , below which performance is known to degrade [Spigler et al., 2018]).”
A jamming transition from under- to over-parametrization affects loss landscape and generalization (2018, background): “dynamics [29, 2] support that the landscape is characterized by an abundance of flat directions, even near its bottom, at odds with traditional glassy systems.”
Trainability and Accuracy of Artificial Neural Networks: An Interacting Particle System Approach (2018, background): “Several recent papers have focused on the analysis of the shape and properties of the ob-jective or “loss” function landscape [5,13,29].”
Dynamics of stochastic gradient descent for two-layer neural networks in the teacher–student setup (2019, background): “[28] Marco Baity-Jesi, Levent Sagun, Mario Geiger, Stefano Spigler, Gérard Ben Arous, Chiara Cammarota, Yann LeCun, Matthieu Wyart, and Giulio Biroli.”