An Investigation into Neural Net Optimization via Hessian Eigenvalue Density

By Behrooz Ghorbani, Shankar Krishnan, Ying Xiao. 2019

460 citations recorded by the source corpus.

To understand the dynamics of optimization in deep neural networks, we develop a tool to study the evolution of the entire Hessian spectrum throughout the optimization process. Using this, we study a number of hypotheses concerning smoothness, curvature, and sharpness in the deep learning literature. We then thoroughly analyze a crucial structural feature of the spectra: in non-batch normalized networks, we observe the rapid appearance of large isolated eigenvalues in the spectrum, along with a surprising concentration of the gradient in the corresponding eigenspaces. In batch normalized networks, these two effects are almost absent. We characterize these effects, and explain how they affect optimization speed through both theory and experiments. As part of this work, we adapt advanced tools from numerical linear algebra that allow scalable and accurate estimation of the entire Hessian spectrum of ImageNet-scale neural networks; this technique may be of independent interest in other applications.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

Scaling Laws for Neural Language Models (2020, neutral): “Making this connection quantitative will require a characterization of the Hessian spectrum [Pap18, GKX19, GARD18].”
Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model (2019, positive): “This closely matches the estimated eigenspectrum of the Hessian of a convolutional network (see Figure 9 and Appendix E.4), and is also consistent with recent work finding heavy tailed eigenspectra of neural network Hessians [Ubaru et al., 2017, Ghorbani et al., 2019].”
Sharpness-Aware Minimization for Efficiently Improving Generalization (2020, methodology): “Due to the parameter space’s dimensionality, we approximate the Hessian spectrum using the Lanczos algorithm of Ghorbani et al. (2019).”
The jamming transition as a paradigm to understand the loss landscape of deep neural networks (2018, background): “[73] Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao,”
PyHessian: Neural Networks Through the Lens of the Hessian (2019, methodology): “Moreover, the work of [9] measured the Hessian ESD, based on the stochastic Lanczos”
Deep Learning Through the Lens of Example Difficulty (2021, background): “Examples of such work include studies of eigenvalues of the Hessian and the geometry of the loss landscape (Ghorbani et al., 2019; Yao et al., 2020; Sagun et al., 2016; Li et al., 2018; Pennington and Bahri, 2017; Sagun et al., 2018), studies of margin and effective generalization measures (Long and”
Stiffness: A New Perspective on Generalization in Neural Networks (2019, background): “am matrix to study convergence in neural network empirical loss. Pennington & Worah (2017) study the concentration properties of a similar covariance matrix formed from the output of the network. Ghorbani et al. (2019) investigate the Hessian eigenspectrum and Papyan (2019) show how it is relate”
Gradient Flow in Sparse Neural Networks and How Lottery Tickets Win (2020, background): “When the gradient is zero, or uninformative due to the error term of the approximation, analyzing the Hessian could provide additional insights (Ghorbani et al., 2019; Papyan, 2019; Sagun et al., 2017).”
The Full Spectrum of Deepnet Hessians at Scale: Dynamics with SGD Training and Sample Size. (2018, background): “Ghorbani et al. [2019] do not remove the reorthogonalization step.”
Fine-Tuning Language Models with Just Forward Passes (2023, background): “However, many previous works have shown that the Hessian of the loss for deep neural networks trained by SGD has remarkably low effective rank [74, 75, 36, 107, 105, 82].”
The Break-Even Point on Optimization Trajectories of Deep Neural Networks (2020, methodology): “Ghorbani et al. (2019) show that without batch normalization, mini-batch gradients are largely contained in the subspace spanned by the top eigenvectors of noncentered K .”
Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability (2021, background): “…Finally, our precise characterization of the behavior of the sharpness during full-batch gradient descent adds to a growing body of work that empirically investigates the Hessian spectrum of neural networks (Sagun et al., 2017; Ghorbani et al., 2019; Li et al., 2020a; Papyan, 2018; 2019; 2020).”
How Do Vision Transformers Work? (2022, methodology): “Ghorbani et al. (2019) visualized the Hessian eigenvalue spectrum by using the Lanczos quadrature algorithm for full batch.”
The Early Phase of Neural Network Training (2020, result): “These results have been largely confirmed in large scale studies (Ghorbani et al., 2019), but note they depend to some extent on architecture and (absence of) batch normalization (Ioffe & Szegedy, 2015).”
MetaInit: Initializing learning by learning to initialize (2019, background): “Most recently, it has been observed [51, 46, 18] that Hessian conditioning is intimately related to favorable learning dynamics in deep networks.”
Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisited (2020, background): “Similarly, the empirical observations of Papyan (2018), Sagun et al. (2017), and Ghorbani et al. (2019) demonstrate that even for extremely wide neural networks, the Hessian is not constant through training.”