On Natural Learning and Pruning in Multilayered Perceptrons

By Tom M. Heskes. 2000

97 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computational Mechanics, Model Architectures, Optimization & Training, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Model Architectures

In Optimization & Training

In Signal Processing

Citations in the corpus

Revisiting Natural Gradient for Deep Networks (2013, background): “While Heskes (2000) precedes both Schraudolph (2002); Martens (2010) both Hessian Free and Krylov Subspace Descent are introduced as purely approximations to second order methods.”
Optimizing Neural Networks with Kronecker-factored Approximate Curvature (2015, methodology): “The work most closely related to ours is that of Heskes (2000), who proposed an approximation of the Fisher of feed-forward neural networks similar to our Kronecker-factored block-diagonal approximation ˘ F from Section 3.2, and used it to derive an efﬁcient approximate natural-gradient based…”
New Insights and Perspectives on the Natural Gradient Method (2014, methodology): “Heskes (2000) showed that the Fisher and the classical Gauss-Newton matrix are equivalent in the case of the squared error loss and proposed using the Fisher as an approximation to the Hessian in more general contexts. Concurrently with this work, Pascanu and Bengio (2014) showed that for several co”
A Kronecker-factored approximate Fisher matrix for convolution layers (2016, methodology): “Our procedure is perhaps more closely analogous to earlier Kronecker product-based natural gradient approximations (Heskes, 2000; Povey et al., 2015) than to K-FAC itself.”
Three Mechanisms of Weight Decay Regularization (2018, methodology): “It has been shown that the GN matrix is equivalent to the Fisher matrix in the case of regression task with squared error loss (Heskes, 2000). However, they are not identical for the case of classification, where cross-entropy loss is commonly used. Nevertheless, Martens (2014) showed that the Fishe”
WoodFisher: Efficient second-order approximations for model compression (2020, methodology): “26 3.4.1 Suitability of the Empirical Fisher . . . . . . . . . . . . . . . . . . . . . . . 26 3.4.2 Issues with using K-FAC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 3.5 Other approximation methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30”
Natural conjugate gradient training of multilayer perceptrons (2006, methodology): “We can further reduce the complexity assuming [5] independence between the output ok of unit k at a given layer and the generalized error δl of unit l of the next layer.”
WoodFisher: Efficient Second-Order Approximation for Neural Network Compression (2020, methodology): “Kronecker-factorization (K-FAC) methods [15, 37] replace the expectation of a Kronecker product between two matrices (that arises in the formulation of Fisher blocks between two layers) as the Kronecker product between the expectations of two matrices.”
Gradient-Based Meta-Learning with Learned Layerwise Metric and Subspace (2018, background): “Previous works (Heskes, 2000; Martens & Grosse, 2015; Desjardins et al., 2015) have used such an approximation of the curvature of a neural network.”
Limitations of the Empirical Fisher Approximation (2019, methodology): “2 Heskes [2000] showed this for regression with squared loss, Pascanu and Bengio [2014] for classification with cross-entropy loss, and Martens [2014] for general exponential families.”
Kronecker-factored Curvature Approximations for Recurrent Neural Networks (2018, background): “This is the basic Kronecker factored approximation (Heskes, 2000; Martens & Grosse, 2015; Povey et al., 2015), which is related to the approximation made in the Natural Neural Nets approach (Desjardins et al.”
Exact natural gradient in deep linear networks and its application to the nonlinear case (2018, methodology): “10 ; see also Heskes [2000]; Martens and Grosse [2015]).”