Negative eigenvalues of the Hessian in deep neural networks

By Guillaume Alain, Nicolas Le Roux, Pierre-Antoine Manzagol. 2018

47 citations recorded by the source corpus.

The loss function of deep networks is known to be non-convex but the precise nature of this nonconvexity is still an active area of research. In this work, we study the loss landscape of deep networks through the eigendecompositions of their Hessian matrix. In particular, we examine how important the negative eigenvalues are and the benefits one can observe in handling them appropriately.

Subfields: Artificial Intelligence, Computational Mechanics, Computational Theory and Mathematics, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Computational Theory and Mathematics

In Optimization & Training

Citations in the corpus

A Closer Look at the Optimization Landscapes of Generative Adversarial Networks (2019, methodology): “Another way to gain insight into the landscape of deep neural networks is by looking at the Hessian of the loss; this was done in the context of single objective minimization by (Dauphin et al., 2014; Sagun et al., 2016, 2017; Alain et al., 2019).”
The Break-Even Point on Optimization Trajectories of Deep Neural Networks (2020, methodology): “Following He et al. (2015), we regularize the model using weight decay 0 .”
Lightlike Neuromanifolds, Occam's Razor and Deep Learning (2019, background): “…a local optimum, and in the second order term, − N J (ˆΘ) is the Hessian matrix of the likelihood function log p ( X | Θ) evaluated at ˆΘ , as we have Notice that J (ˆΘ) (cid:23) 0 , while in general the Hessian of the loss of a DNN evaluated at Θ (cid:54) = ˆΘ can have a negative spectrum [2, 40].”
Interpreting Adversarial Robustness: A View from Decision Surface in Input Space (2018, background): “Meanwhile, magnitude of eigenvalues λi of Hessian denotes the curvature (Alain et al. (2018); Cheeger & Ebin (2008)), which is defined as: κ = 1/r. (10) κ is the reciprocal of the radius R of osculating circle at current point.”