Estimating the Hessian by Back-propagating Curvature

By James Martens, Ilya Sutskever, Kevin Swersky. 2012

87 citations recorded by the source corpus.

In this work we develop Curvature Propagation (CP), a general technique for efficiently computing unbiased approximations of the Hessian of any function that is computed using a computational graph. At the cost of roughly two gradient evaluations, CP can give a rank-1 approximation of the whole Hessian, and can be repeatedly applied to give increasingly precise unbiased estimates of any or all of the entries of the Hessian. Of particular interest is the diagonal of the Hessian, for which no general approach is known to exist that is both efficient and accurate. We show in experiments that CP turns out to work well in practice, giving very accurate estimates of the Hessian of neural networks, for example, with a relatively small amount of work. We also apply CP to Score Matching, where a diagonal of a Hessian plays an integral role in the Score Matching objective, and where it is usually computed exactly using inefficient algorithms which do not scale to larger and more complex models.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

In Optimization & Training

Citations in the corpus

No more pesky learning rates (2012, methodology): “We propose three variants on this spectrum: vSGD-l uses local gradient variance terms and the local diagonal Hessian estimates, leading to Comparison between SGD for two diﬀerent ﬁxed learning rates 1.0 and 0.2, and two cooling schedules η = 1 /t and η = 0 .”
Improved multitask learning through synaptic intelligence (2017, methodology): “However, in contrast to EWC, here we are putting forward a method which allows for online computation of the importance measure, whereas EWC relies on the diagonal of the Fisher information metric at the ﬁnal parameters, which has to be computed during a separate phase at the end of each task.”
YellowFin and the Art of Momentum Tuning (2017, methodology): “As future work, we believe that more accurate curvature estimation methods, like the bbprop method (Martens et al., 2012) can further improve YELLOWFIN.”
Sliced Score Matching: A Scalable Approach to Density and Score Estimation (2019, methodology): “Moreover, Martens et al. (2012) argues from a theoretical perspective that it is unlikely that there exists an algorithm for computing the diagonal of the Hessian deﬁned by an arbitrary computation graph within a constant number of forward and backward passes.”
Deep Energy Estimator Networks (2018, methodology): “However, as discussed above, such an approach is often considered to be computationally prohibitive due to the appearance of second-order derivatives in the objective function (Kingma & LeCun, 2010) and one must resort to approximations (Martens et al., 2012a).”
RMSProp and equilibrated adaptive learning rates for non-convex optimization (2015, neutral): “The variance of this estimator for an element i is j H 2 ji − H 2 ii , while the method in Martens et al. (2012) has H 2 ii .”
Exponential convergence rates for Batch Normalization: The power of length-direction decoupling in non-convex optimization (2018, background): “This does not only make gradient based training easier but also suggests the possibility of using partial second order information, such as diagonal Hessian approximations (e.g. proposed in (Martens et al., 2012)).”
Neural Networks and Deep Learning (2018, unknown): “The second-order derivatives of the loss function can be computed with backpropagation [315], although this is rarely done in practice.”
How to Train Your Energy-Based Models (2021, methodology): “Although SM only requires the trace of the Hessian, it is still expensive to compute even with modern hardware and automatic diﬀerentiation packages (Martens et al., 2012).”
Efficient Learning of Generative Models via Finite-Difference Score Matching (2020, methodology): “[58], we adopt the NICE [8] model and train it by minimizing the Fisher divergence using different approaches including approximate back-propagation (Approx BP) [25] and curvature propagation (CP) [39].”
Continual Learning Through Synaptic Intelligence (2017, background): “The Hessian is one possible metric, but it can be negative definite and computing even the diagonal adds additional overhead over standard backpropagation (Martens et al., 2012).”
Pruning Convolutional Neural Networks for Resource Efficient Inference (2016, unknown): “Several researchers tried to tackle this problem with approximation techniques (Martens, 2010; Martens et al., 2012).”
Improving DNN Robustness to Adversarial Attacks using Jacobian Regularization (2018, background): “As shown in [29], the decision boundary between two classes k 1 and k 2 can be locally referred to as the hyper-surface Relying on the work in [32] let us use the approximation H k ( x ) ≈ J k ( x ) T J k ( x ) where J k ( x ) is the k th row in the matrix J ( x ).”
Stochastic Gradient Descent: Going As Fast As Possible But Not Faster (2017, background): “zation methods are adversely affected. Hessian-based approaches, which optimally handle convex optimization problems however ill-conditioned they are, do not scale up and approximations are required [Martens et al., 2012]. Overall, Stochastic Gradient Descent (SGD) is increasingly adopted both in co”