Empirical Analysis of the Hessian of Over-Parametrized Neural Networks

By Levent Sagun, Utku Evci, V. U. Güney, Yann Dauphin, L. Eon Bottou. 2017

516 citations recorded by the source corpus.

We study the properties of common loss surfaces through their Hessian matrix. In particular, in the context of deep learning, we empirically show that the spectrum of the Hessian is composed of two parts: (1) the bulk centered near zero, (2) and outliers away from the bulk. We present numerical evidence and mathematical justifications to the following conjectures laid out by Sagun et al. (2016): Fixing data, increasing the number of parameters merely scales the bulk of the spectrum; fixing the dimension and changing the data (for instance adding more clusters or making the data less separable) only affects the outliers. We believe that our observations have striking implications for non-convex optimization in high dimensions. First, the flatness of such landscapes (which can be measured by the singularity of the Hessian) implies that classical notions of basins of attraction may be quite misleading. And that the discussion of wide/narrow basins may be in need of a new perspective around over-parametrization and redundancy that are able to create large connected components at the bottom of the landscape. Second, the dependence of small number of large eigenvalues to the data distribution can be linked to the spectrum of the covariance matrix of gradients of model outputs. With this in mind, we may reevaluate the connections within the data-architecture-algorithm framework of a model, hoping that it would shed light into the geometry of high-dimensional and non-convex spaces in modern applications. In particular, we present a case that links the two observations: small and large batch gradient descent appear to converge to different basins of attraction but we show that they are in fact connected through their flat region and so belong to the same basin.

Subfields: Artificial Intelligence, Computational Mechanics, Statistics and Probability, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Statistics and Probability

In Optimization & Training

Citations in the corpus

An Investigation into Neural Net Optimization via Hessian Eigenvalue Density (2019, background): “Sagun et al. [23, 24] study the spectrum of the Hessian for small two-layer feed-forward networks.”
Scaling description of generalization with number of parameters in deep learning (2019, background): “Numerical and theoretical studies [6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17] show that in the overparametrized regime, the loss landscape of DNNs is not rough with isolated minima as initially thought [18, 19], but instead has connected level sets and presents many flat directions, even near its g”
The Full Spectrum of Deep Net Hessians At Scale: Dynamics with Sample Size (2018, background): “As pointed out by [14, 15], negative eigenvalues exist in the spectrum of the train Hessian.”
Gradient Descent Happens in a Tiny Subspace (2018, background): “In Sagun et al. (2016; 2017), the spectrum of the Hessian for deep learning cross-entropy losses was analyzed in depth.”
A Closer Look at the Optimization Landscapes of Generative Adversarial Networks (2019, methodology): “Another way to gain insight into the landscape of deep neural networks is by looking at the Hessian of the loss; this was done in the context of single objective minimization by (Dauphin et al., 2014; Sagun et al., 2016, 2017; Alain et al., 2019).”
The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learning (2017, background): “While we do not yet understand why these interpolated classifiers generalize so well to unseen data, there is ample empirical evidence for their excellent generalization performance in deep neural networks (Gupta et al., 2015; Chaudhari et al., 2016; Zhang et al., 2016; Huang et al., 2016; Sagun et ”
The jamming transition as a paradigm to understand the loss landscape of deep neural networks (2018, methodology): “[50] A similar decomposition has been used in [25, 77, 78].”
Generalization Guarantees for Neural Networks via Harnessing the Low-rank Structure of the Jacobian (2019, background): “Generalization properties of stochastic gradient descent is extensively studied empirically [34, 28, 50, 16, 29, 24, 25].”
A Simple Baseline for Bayesian Uncertainty in Deep Learning (2019, neutral): “[57] Sagun, L.”
Learning explanations that are hard to vary (2020, neutral): “1 in [Sagun et al., 2017].”
Online Structured Laplace Approximations For Overcoming Catastrophic Forgetting (2018, methodology): “At the same time, even though the Laplace method is a local approximation, we would expect it to leave sufﬁcient ﬂexibility for the parameters to adapt to new tasks, as the Hessian of neural networks has been observed to be ﬂat in most directions [36].”
Essentially No Barriers in Neural Network Energy Landscape (2018, neutral): “A similar view had previously been conjectured by (Sagun et al., 2017).”
Three Factors Influencing Minima in SGD (2017, background): “In the work of Sagun et al. (2017), the Hessian is also related to the gradient covariance matrix, and both are found to be highly anisotropic.”
There Are Many Consistent Explanations of Unlabeled Data: Why You Should Average (2018, background): “We expect width along random rays to be less meaningful because there are many directions in the parameter space that do not change the network outputs (Dinh et al., 2017b; GurAri et al., 2018; Sagun et al., 2017).”
To understand deep learning we need to understand kernel learning (2018, background): “, 2016) which fit the data exactly (only the classification error is reported there, other references also report MSE (Chaudhari et al., 2016; Huang et al., 2016; Sagun et al., 2017; Bartlett et al., 2017)).”
A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networks (2019, background): “A final remark on the issues of this approach is the observation that landscape is flat at the bottom regardless of the batch size used in SGD [46].”