Emergent properties of the local geometry of neural loss landscapes
By Stanislav Fort, Surya Ganguli. 2019
64 citations recorded by the source corpus.
The local geometry of high dimensional neural network loss landscapes can both challenge our cherished theoretical intuitions as well as dramatically impact the practical success of neural network training. Indeed recent works have observed 4 striking local properties of neural loss landscapes on classification tasks: (1) the landscape exhibits exactly $C$ directions of high positive curvature, where $C$ is the number of classes; (2) gradient directions are largely confined to this extremely low dimensional subspace of positive Hessian curvature, leaving the vast majority of directions in weight space unexplored; (3) gradient descent transiently explores intermediate regions of higher positive curvature before eventually finding flatter minima; (4) training can be successful even when confined to low dimensional {\it random} affine hyperplanes, as long as these hyperplanes intersect a Goldilocks zone of higher than average curvature. We develop a simple theoretical model of gradients and Hessians, justified by numerical experiments on architectures and datasets used in practice, that {\it simultaneously} accounts for all $4$ of these surprising and seemingly unrelated properties. Our unified model provides conceptual insights into the emergence of these properties and makes connections with diverse topics in neural networks, random matrix theory, and spin glasses, including the neural tangent kernel, BBP phase transitions, and Derrida's random energy model.
Subfields: Artificial Intelligence, Statistics and Probability
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Kyunghyun Cho
- Surya Ganguli
- Stanislaw Jastrzebski
- Stanislav Fort
- Devansh Arpit
- Mohit Bansal
- Quoc V. Le
In Statistics and Probability
- Kyunghyun Cho
- Devansh Arpit
- Gintare Karolina Dziugaite
- Surya Ganguli
- Stanislav Fort
- Stanislaw Jastrzebski
- Madhu S. Advani
Citations in the corpus
- Linear Mode Connectivity in Multitask and Continual Learning (2020, neutral): “We note that similar to the observation by Fort & Ganguli (2019) and Mirzadeh et al. (2020), the eigenspectrum of the Hessian includes a bulk plus C larger outlier eigenvalues where C is the number of classes.”
- The Break-Even Point on Optimization Trajectories of Deep Neural Networks (2020, methodology): “Papyan (2019); Fort & Ganguli (2019) show that the top eigenvalues of H emerge due to clustering of gradients of the logits.”
- Deep Learning on a Data Diet: Finding Important Examples Early in Training (2021, result): “Fort and Ganguli [12, Sec.”