The Full Spectrum of Deep Net Hessians At Scale: Dynamics with Sample Size

By Vardan Papyan. 2018

36 citations recorded by the source corpus.

Previous works observed the spectrum of the Hessian of the training loss of deep neural networks. However, the networks considered were of minuscule size. We apply state-of-the-art tools in modern high-dimensional numerical linear algebra to approximate the spectrum of the Hessian of deep nets with tens of millions of parameters. Our results corroborate previous findings, based on small-scale networks, that the Hessian exhibits 'spiked' behavior, with several outliers isolated from a continuous bulk. However we find that the bulk does not follow a simple Marchenko-Pastur distribution, as previously suggested, but rather a heavier-tailed distribution. Finally, we document the dynamics of the outliers and the bulk with varying sample size.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yoshua Bengio
2. Vardan Papyan
3. Behrooz Ghorbani
4. Shankar Krishnan
5. Ying Xiao
6. Guillaume Alain
7. Kyunghyun Cho
8. David L. Donoho

In Computer Vision and Pattern Recognition

1. Yoshua Bengio
2. Luc Van Gool
3. Dacheng Tao
4. Ming-Hsuan Yang
5. Xiaogang Wang
6. Trevor Darrell
7. Jitendra Malik
8. Andrew Zisserman

In Statistical and Nonlinear Physics

1. Shankar Krishnan
2. Ying Xiao
3. Behrooz Ghorbani
4. David L. Donoho
5. Guillaume Alain
6. Vardan Papyan
7. Guido Montúfar

In Scaling & Learning Theory

1. Behrooz Ghorbani
2. Shankar Krishnan
3. Ying Xiao
4. Pierre-Antoine Manzagol
5. Guillaume Alain
6. Nicolas Le Roux
7. Vardan Papyan

Citations in the corpus

• Scaling Laws for Neural Language Models (2020, neutral): “Making this connection quantitative will require a characterization of the Hessian spectrum [Pap18, GKX19, GARD18].”
• An Investigation into Neural Net Optimization via Hessian Eigenvalue Density (2019, methodology): “During the preparation of this paper, [21] appeared on Arxiv which briefly introduces the same spectrum estimation methodology and studies the Hessian on small subsamples of MNIST and CIFAR-10 at the end of the training.”
• Negative eigenvalues of the Hessian in deep neural networks (2018, background): “In addition to the work of Dauphin et al. (2014), Papyan (2018) studied the spectrum of the Hessian and other matrices at a stationary point while Sagun et al. (2016) analyzed the evolution of the spectrum of the Hessian during the optimization.”
• Prevalence of neural collapse during the terminal phase of deep learning training (2020, background): “(41–43) corroborated these findings at scale on modern deep networks and large datasets.”