The Full Spectrum of Deepnet Hessians at Scale: Dynamics with SGD Training and Sample Size.

120 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisited (2020, background): “Similarly, the empirical observations of Papyan (2018), Sagun et al. (2017), and Ghorbani et al. (2019) demonstrate that even for extremely wide neural networks, the Hessian is not constant through training.”
Fine-Tuning Language Models with Just Forward Passes (2023, background): “However, many previous works have shown that the Hessian of the loss for deep neural networks trained by SGD has remarkably low effective rank [74, 75, 36, 107, 105, 82].”
Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability (2021, background): “…Finally, our precise characterization of the behavior of the sharpness during full-batch gradient descent adds to a growing body of work that empirically investigates the Hessian spectrum of neural networks (Sagun et al., 2017; Ghorbani et al., 2019; Li et al., 2020a; Papyan, 2018; 2019; 2020).”
Negative eigenvalues of the Hessian in deep neural networks (2018, background): “In addition to the work of Dauphin et al. (2014), Papyan (2018) studied the spectrum of the Hessian and other matrices at a stationary point while Sagun et al. (2016) analyzed the evolution of the spectrum of the Hessian during the optimization.”