PyHessian: Neural Networks Through the Lens of the Hessian

By Zi-Mei Yao, A. Gholami, Kurt Keutzer, Michael W. Mahoney. 2019

401 citations recorded by the source corpus.

We present PYHESSIAN, a new scalable framework that enables fast computation of Hessian (i.e., second-order derivative) information for deep neural networks. PYHESSIAN enables fast computations of the top Hessian eigenvalues, the Hessian trace, and the full Hessian eigenvalue/spectral density; it supports distributed-memory execution on cloud/supercomputer systems; and it is available as open source [1]. This general framework can be used to analyze neural network models, including the topology of the loss landscape (i.e., curvature information) to gain insight into the behavior of different models/optimizers. As an example, we analyze the effect of residual connections and Batch Normalization layers on the trainability of neural networks. One recent claim, based on simpler first-order analysis, is that residual connections and Batch Normalization make the loss landscape "smoother," thus making it easier for Stochastic Gradient Descent to converge to a good solution. Our second-order analysis, easily enabled by PYHESSIAN, shows new finer-scale insights, demonstrating that while conventional wisdom is sometimes validated, in other cases it is simply incorrect. In particular, we find that Batch Normalization does not necessarily make the loss landscape smoother, especially for shallow networks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

Citations in the corpus

G-DAug: Generative Data Augmentation for Commonsense Reasoning (2020, methodology): “Specifically, using the fact that a larger Hessian trace for a model implies a sharper local minimum (Yao et al., 2019), we compute the Hessian trace of 10 baseline and 10 G-DA UG c -Combo methods using LR (q/a) 5e-5/5e-5 2e-5/5e-5 2e-5/5e-5 2e-5/5e-5 1e-5/5e-5 Epochs (q/a) 8/12 6/6 3/3 3/3 3/1…”
Deep Learning Through the Lens of Example Difficulty (2021, background): “Examples of such work include studies of eigenvalues of the Hessian and the geometry of the loss landscape (Ghorbani et al., 2019; Yao et al., 2020; Sagun et al., 2016; Li et al., 2018; Pennington and Bahri, 2017; Sagun et al., 2018), studies of margin and effective generalization measures (Long and”
Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training (2023, neutral): “The loss functions of modern deep learning problems often have different curvatures across different parameter dimensions (Sa-gun et al., 2016; Ghorbani et al., 2019; Zhang et al., 2020; Yao et al., 2020).”
SqueezeLLM: Dense-and-Sparse Quantization (2023, background): “While it is possible to compute Hessian information with a second backprop [77], this approach can be costly for LLMs due to the increased memory requirements.”
Normalization Techniques in Training DNNs: Methodology, Analysis and Application (2020, background): “It is disputed by the empirical studies in [202], showing the exact opposite behavior of BN on a ResNet-20 network.”
ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learning (2020, background): “Some examples include Hessian based model compression [16, 17, 21, 25], adversarial attacks [59], and studies of the loss landscape topology for different NN architectures [44, 57], just to name a few.”
Stabilizing Transformer Training by Preventing Attention Entropy Collapse (2023, methodology): “…we computed the sharpness – the largest singular value of the Hessian (the second order derivative of the loss with respect to the model parameters), as its magnitude has implications for training stability (Ghorbani et al., 2019; Yao et al., 2020; Cohen et al., 2021; 2022; Gilmer et al., 2021).”
Batch normalization provably avoids ranks collapse for randomly initialised deep networks (2020, background): “Furthermore, even more recent empirical studies again dispute these ﬁndings, by observing the exact opposite behaviour of B N on a ResNet20 network [34].”
Fine-Tuning Language Models with Just Forward Passes (2023, background): “[100] Zhewei Yao, Amir Gholami, Kurt Keutzer, and Michael W Mahoney.”
Structured Pruning for Deep Convolutional Neural Networks: A Survey (2023, background): “This approximation costs the same as backpropagating gradients [242].”
Characterizing possible failure modes in physics-informed neural networks (2021, background): “This tends to be more informative than perturbing the model parameters in random directions [41, 42].”
HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networks (2019, background): “computational cost [31], but we show this is not an issue here.”
How Do Vision Transformers Work? (2022, methodology): “We use the implementation of Yao et al. (2020) to this end.”
A Theory on Adam Instability in Large-Scale Machine Learning (2023, background): “The spectrum of the hessian of a neural network loss function has been the focus of various studies [Sankar et al., 2021, Liao and Mahoney, 2021, Yao et al., 2020].”