Fast Curvature Matrix-Vector Products for Second-Order Gradient Descent

By N. Schraudolph. 2002

423 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Optimization & Training, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

In Optimization & Training

In Signal Processing

Citations in the corpus

Revisiting Natural Gradient for Deep Networks (2013, methodology): “Hessian-Free as well as Krylov Subspace Descent rely on the extended Gauss-Newton approximation of the Hessian, G N , instead of the actual Hessian (see Schraudolph (2002)): ∂ r 2 ∂ r ∂θ The last step of eq.”
No more pesky learning rates (2012, methodology): “Stochastic meta-descent (SMD, Schraudolph (1999; 2002)) uses a related multiplicative update of learning rates.”
Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model (2019, positive): “Consistent with the empirical results of Shallue et al. [2018], each optimizer shows two distinct regimes: a small-batch (stochastic) regime with perfect linear scaling, and a large-batch (deterministic) regime insensitive to batch size. We call the phase transition between these regimes the critica”
Neural Optimizer Search with Reinforcement Learning (2017, methodology): “…are difﬁcult and slow to train, and many methods have been designed to reduce this difﬁculty (e.g., Riedmiller & Braun (1992); LeCun et al. (1998); Schraudolph (2002); Martens (2010); Le et al. (2011); Duchi et al. (2011); Zeiler (2012); Martens & Sutskever (2012); Schaul et al. (2013); Pascanu &…”
Deep learning in neural networks: An overview (2014, methodology): “19 2011: Hessian-Free Optimization for RNNs Also in 2011 it was shown (Martens and Sutskever, 2011) that Hessian-free optimization (e.g., Møller, 1993; Pearlmutter, 1994; Schraudolph, 2002) (Sec.”
Continual Learning With Extended Kronecker-Factored Approximate Curvature (2020, positive): “The quadratic penalty loss can prevent the parameters from moving too far from the initial point when positive semidefinite approximations of Hessian such as the Fisher information matrix [28] or the generalized Gauss–Newton matrix [33] are used; however, the problem can be still illposed.”
Uncertainty-guided Continual Learning with Bayesian Neural Networks (2019, methodology): “N-VCL (Tseran et al., 2018) uses a Gauss-Newton approximation introduced by (Schraudolph, 2002; Graves, 2011) to estimate the VCL objective function and used natural gradient method proposed in (Khan et al., 2018) to exploit the Riemannian geometry of the variational posterior by scaling the…”
New Insights and Perspectives on the Natural Gradient Method (2014, methodology): “An approximation of the Fisher known as the “empirical Fisher” (Schraudolph, 2002), which we denote by F̄ , is commonly used in practical natural gradient methods.”
Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation (2017, methodology): “Trust-region policy optimization (TRPO) [22] avoids explicitly storing and inverting the Fisher matrix by using Fisher-vector products [21].”
A Kronecker-factored approximate Fisher matrix for convolution layers (2016, methodology): “Conjugate gradient (CG) is defined in terms of matrix-vector products Fv, which can be computed efficiently and exactly using the method outlined by Schraudolph (2002).”
Krylov Subspace Descent for Deep Learning (2011, methodology): “As described in [13], it is possible to efficiently multiply a vector byG using a version of the “Pearlmutter trick”; the algorithm is similar in spirit to b ackprop and we give it as Algorithm 1.”
Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adam (2018, methodology): “To avoid negative variances in the VON update, we propose to use the Generalized Gauss-Newton (GGN) approximation (Schraudolph, 2002; Martens, 2014; Graves, 2011):”
Large-Scale Feature Learning With Spike-and-Slab Sparse Coding (2012, methodology): “We implement conjugate gradient descent efficiently by using the R-operator to perform Hessianvector products rather than computing the entire Hessian explicitly (Schraudolph, 2002).”
Variable Metric Stochastic Approximation Theory (2009, methodology): “Therefore, online optimization methods that work with small subsamples of training data have received considerable attention recently (Kivinen and Warmuth, 1997; Schraudolph, 2002; Azoury and Warmuth, 2001; Shalev-Shwartz et al., 2007; Schraudolph et al., 2007).”
Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks (2006, neutral): “in use for neural networks (LeCun et al., 1998; Schraudolph, 2002 ).”
Multimodal integration learning of robot behavior using deep neural networks (2014, background): “The name “Hessian-free” indicates that the CG does not necessarily require the costly explicit Hessian matrix; instead, the matrix-vector product between the Hessian matrix H or the Gauss-Newton matrix G and gradient vector p is sufficient (for more details on the concrete implementation, see [28], ”
Learning Implicit Generative Models with the Method of Learned Moments (2018, methodology): “When using only gradient features, one can speed up gradient computation by ∼ 20% by using a Hessian-vector product-like trick (Pearlmutter, 1994; Schraudolph, 2002).”