Exact Calculation of the Hessian Matrix for the Multilayer Perceptron

By Charles M. Bishop. 1992

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Bayesian Methods for Adaptive Models (2011, background): “I expect an exact analytic evaluation of the second derivatives [9] would resolve this.”
Training Deep and Recurrent Networks with Hessian-Free Optimization (2012, background): “In case of the Gauss-Newton matrix, it is possible to obtain the exact diagonal at the cost of k runs of backpropagation, where k is the number of output units [6].”
Fast Exact Multiplication by the Hessian (1994, background): “…(the matrix of second derivative terms ¶ 2 E (cid:26) ¶ w i ¶ w j of the error E with respect to the weights (cid:9) ) of a backpropagation network (Bishop, 1992; Werbos, 1992; Buntine and Weigend, 1991), or reasonable estimates thereof (MacKay, 1991)—but even storing the full Hessian is…”
Non-iterative Knowledge Fusion in Deep Convolutional Neural Networks (2018, methodology): “e intervals, improving backpropagation algorithm etc. (Buntine & Weigend, 1994; Pascanu & Bengio, 2013). The exact calculation of Hessian matrix for the multilayer perceptron was performed by Bishop (1992). However, to use standard machine learning software packages, it is better to simplify”
Pattern Recognition and Machine Learning (2006, unknown): “The Hessian can also be evaluated exactly, for a network of arbitrary feed-forward topology, using extension of the technique of backpropagation used to evaluate first derivatives, which shares many of its desirable features including computational efficiency (Bishop, 1991; Bishop, 1992).”
Exact Calculation of the Product of the Hessian Matrix of Feed-Forward Network Error Functions and a Vector in 0(N) Time (1993, methodology): “Several researchers have recently derived formulae for exact calculation of the elements in the Hessian matrix [Buntine 91], [Bishop 92].”
Understanding Convolutional Neural Networks (2014, unknown): “5 can be found in [Bis92].”
Ensemble learning in Bayesian neural networks (1998, methodology): “which is the form of covariance used in factor analysis (Bishop 1998a). This reduces the number of independent parameters in the covariance matrix from k(k + 1)/2 to k(s + 1) − s(s + 1)/2, which is now linear in k. Thus, the number of parameters can be controlled by changing s and, unlike a diagonal”
The power of quantum neural networks (2020, background): “The Hessian quantifies the curvature of a model’s loss function at a point in its parameter space [30].”