Exact Calculation of the Product of the Hessian Matrix of Feed-Forward Network Error Functions and a Vector in 0(N) Time
By Martin Fodslette Møller. 1993
Several methods for training feed-forward neural networks require second order information from the Hessian matrix of the error function. Although it is possible to calculate the Hessian matrix exactly it is often not desirable because of the computation and memory requirements involved. Some learning techniques do, however, only need the Hessian matrix times a vector. This paper presents a method to calculate the Hessian matrix times a vector in O(N) time, where N is the number of variables in the network. This is the same order as the calculation of the gradient to the error function. The usefulness of this algorithm is demonstrated by improvement of existing learning techniques.
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
- N. Schraudolph
- J. Urgen Schmidhuber
- Michael Fairbank
- Yoshua Bengio
- Eduardo Alonso
- James Martens
- Sergey Levine
- Oriol Vinyals
Citations in the corpus
- Deep learning in neural networks: An overview (2014, methodology): “2011: Hessian-free optimization for RNNs Also in 2011 it was shown (Martens & Sutskever, 2011) that Hessian-free optimization (e.g., Møller, 1993; Pearlmutter, 1994; Schraudolph, 2002) (Section 5.6.2) can alleviate the Fundamental Deep Learning Problem (Section 5.9) in RNNs, outperforming standard…”
- Value-gradient learning (2012, methodology): “Here, the backward accumulation is described by Møller (1993), and the derivation takes several pages (dense with lemmas and equations). The forward accumulation is described by Pearlmutter (1994), and the derivation takes just one page to define a differential operator (“R”), which then is used to ”
- Fast Curvature Matrix-Vector Products for Second-Order Gradient Descent (2002, methodology): “This fast Hessian-vector product (Pearlmutter, 1994; Werbos, 1988; Møller, 1993) can be used in conjunction with equation 1.”
- Fast Exact Multiplication by the Hessian (1994, background): “In fact, the Rfbackpropg procedure was independently discovered for that application (Møller, 1993a).”