Singularity of the Hessian in Deep Learning

By Levent Sagun, L. Eon Bottou, Yann LeCun. 2016

59 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

Scaling description of generalization with number of parameters in deep learning (2019, background): “Numerical and theoretical studies [6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17] show that in the overparametrized regime, the loss landscape of DNNs is not rough with isolated minima as initially thought [18, 19], but instead has connected level sets and presents many flat directions, even near its g”
The jamming transition as a paradigm to understand the loss landscape of deep neural networks (2018, background): “Furthermore, studies of the Hessian of the loss function [24–26] and of the learning dynamics [27, 28] support that the landscape is characterized by an abundance of flat directions, even near its bottom, at odds with traditional glassy systems.”
Empirical Analysis of the Hessian of Over-Parametrized Neural Networks (2017, methodology): “More recently, an exact numerical calculation of the Hessian have been carried out by Sagun et al. [2016].”
Sharp Minima Can Generalize For Deep Nets (2017, background): “Since Sagun et al. (2016) seems to suggests that rank deﬁciency in the Hessian is due to over-parametrization of the model, one could conjecture that r − min k ≤ K ( n k ) can be high for thin and deep neural networks, resulting in a majority of large eigenvalues.”
Entropy-SGD: biasing gradient descent into wide valleys (2016, methodology): “For another recent analysis of the Hessian, see the parallel work of Sagun et al. (2016).”
Recovery Guarantees for One-hidden-layer Neural Networks (2017, background): “The recent empirical success of NNs has boosted their theoretical analyses (Feng et al., 2016; Balduzzi, 2016; Bal-duzzi et al., 2016; Sagun et al., 2016; Andoni et al., 2014; Arora et al., 2017; Goel et al., 2017).”
A jamming transition from under- to over-parametrization affects generalization in deep learning (2018, background): “Furthermore, studies of the Hessian of the loss function [12, 13, 14] and of the learning dynamics [15, 16] support that the landscape is characterized by an abundance of flat directions, even near its bottom, at odds with traditional glasses.”
Large Scale Structure of Neural Network Loss Landscapes (2019, background): “One of the ﬁrst studies on curvature were carried out by LeCun et al. [1998], Sagun et al. [2016].”
On the Convex Behavior of Deep Neural Networks in Relation to the Layers' Width (2019, background): “The literature also teaches us about the properties of the Hessian during training (Sagun et al., 2016).”
Comparing dynamics: deep neural networks versus glassy systems (2018, background): “In a related perspective on the loss surface, (Sagun et al., 2016) and (Sagun et al., 2017) demonstrate separate cases where the straight line between two weight conﬁgurations at the bottom of the loss landscape evaluates to the same loss value, in other words there are no barriers between these…”
Deep relaxation: partial differential equations for optimizing deep neural networks (2017, methodology): “Our approach is motivated by experimental evidence in Chaudhari et al. (2016, Figure 1) and Sagun et al. (2016).”
A jamming transition from under- to over-parametrization affects loss landscape and generalization (2018, background): “Furthermore, studies of the Hessian of the loss function [37, 38, 3] and of the learning ar X iv :1 81 0.”
Approximate Fisher Information Matrix to Characterize the Training of Deep Neural Networks (2018, methodology): “Then we discuss other approaches that rely on numerical experiments to measure key aspects of SGD training [5], [6], [7], [8], [24].”
Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapes (2017, background): “The following observations are not unique to this model, shared across different models and datasets (Chaudhari et al., 2017; Sagun et al., 2016).”
On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Length (2018, background): “Sagun et al. (2016) report that the spectral norm of the Hessian reduces towards the end of training.”
Theory of Deep Learning III: explaining the non-overfitting puzzle (2017, background): “The fact that the Hessian is degenerate at minima (many zero eigenvalues) is in fact empirically well-established (for a recent account see (17)).”
The Loss Surface of Deep and Wide Neural Networks (2017, background): “Note however that in practice the Hessian at critical points can be close to singular (at least up to numerical precision), which might affect the training of neural networks negatively (Sagun et al., 2016).”