Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond

By Levent Sagun, L. Eon Bottou, Yann LeCun. 2016

284 citations recorded by the source corpus.

We look at the eigenvalues of the Hessian of a loss function before and after training. The eigenvalue distribution is seen to be composed of two parts, the bulk which is concentrated around zero, and the edges which are scattered away from zero. We present empirical evidence for the bulk indicating how over-parametrized the system is, and for the edges that depend on the input data.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

In Scaling & Learning Theory

Citations in the corpus

An Investigation into Neural Net Optimization via Hessian Eigenvalue Density (2019, background): “Sagun et al. [23, 24] study the spectrum of the Hessian for small two-layer feed-forward networks.”
The Full Spectrum of Deep Net Hessians At Scale: Dynamics with Sample Size (2018, methodology): “[14, 15] introduced the study of the train Hessian and made some initial numerical studies and some initial interpretations on small-scale networks.”
Gradient Descent Happens in a Tiny Subspace (2018, background): “As first explored by Sagun et al. (2016; 2017), the Hessian eigenvalue spectrum appears to naturally separate into “top” and “bulk” components, with the top consisting of the largest k eigenvalues, and the bulk consisting of the rest.”
A Closer Look at the Optimization Landscapes of Generative Adversarial Networks (2019, methodology): “Another way to gain insight into the landscape of deep neural networks is by looking at the Hessian of the loss; this was done in the context of single objective minimization by (Dauphin et al., 2014; Sagun et al., 2016, 2017; Alain et al., 2019).”
Sharp Minima Can Generalize For Deep Nets (2017, background): “Since Sagun et al. (2016) seems to suggests that rank deficiency in the Hessian is due to over-parametrization of the model, one could conjecture that ( r−mink≤K(nk) ) can be high for thin and deep neural networks, resulting in a majority of large eigenvalues.”
Understanding and mitigating gradient pathologies in physics-informed neural networks (2020, methodology): “To obtain a quantitative assessment of stiffness in the gradient flow dynamics of a neural network one can compute and monitor the largest eigenvalue σmax(∇(2)θL(θ)) of the Hessian matrix ∇(2)θL(θ) during model training [31].”
Negative eigenvalues of the Hessian in deep neural networks (2018, background): “In addition to the work of Dauphin et al. (2014), Papyan (2018) studied the spectrum of the Hessian and other matrices at a stationary point while Sagun et al. (2016) analyzed the evolution of the spectrum of the Hessian during the optimization.”
Local Saddle Point Optimization: A Curvature Exploitation Approach (2018, result): “This observation is in accordance with recent empirical evidence [36] showing that the Hessian is actually degenerate for common deep learning architectures.”
Diametrical Risk Minimization: theory and computations (2019, background): “, 2018; Nguyen & Hein, 2017) and that flat minimizers have better generalization properties compared to sharp ones (Chaudhari et al., 2017; Hochreiter & Schmidhuber, 1997; Keskar et al., 2016; Sagun et al., 2016).”
Comparing dynamics: deep neural networks versus glassy systems (2018, background): “In a related perspective on the loss surface, (Sagun et al., 2016) and (Sagun et al., 2017) demonstrate separate cases where the straight line between two weight conﬁgurations at the bottom of the loss landscape evaluates to the same loss value, in other words there are no barriers between these…”
Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisited (2020, background): “Typically as neural networks are trained we observe a gap in the eigenspectrum of the Hessian of the loss (Sagun et al., 2017); a small number of eigenvalues become large while the rest take on values near 0 .”
The Full Spectrum of Deepnet Hessians at Scale: Dynamics with SGD Training and Sample Size. (2018, methodology): “Following the ideas presented in Sagun et al. [2016], we use the Gauss-Newton decomposition of the train (and similarly test) Hessian and write it as a summation of two components:”
Emergent properties of the local geometry of neural loss landscapes (2019, background): “Recent works have observed this phenomenon in small networks [7, 8, 9], as well as large networks [10, 11].”
The Loss Surface of Deep and Wide Neural Networks (2017, background): “Note however that in practice the Hessian at critical points can be close to singular (at least up to numerical precision), which might affect the training of neural networks negatively (Sagun et al., 2016).”