Fast Curvature Matrix-Vector Products

19 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Revisiting Natural Gradient for Deep Networks (2013, methodology): “fferentiate themselves by the approximations they use in order to scale up. One particularly interesting pipeline to scale up such algorithm was originally proposed in Pearlmutter (1994), ﬁnetuned in Schraudolph (2001) and represents the backbone behind both Hessian-Free optimization (Martens, 2010)”
Optimization Methods for Large-Scale Machine Learning (2016, methodology): “e use the term second-order methods, which encompasses a variety of strategies; see x6. We discuss well known 20 inexact Newton and quasi-Newton methods, as well as (generalized) Gauss-Newton methods [14, 141], the natural gradient method [5], and scaled gradient iterations [152,54]. We caution that”
Fast Approximate Natural Gradient Descent in a Kronecker-factored Eigenbasis (2018, background): “…matrix G − 1 yielding the update: θ ← θ − ηG − 1 ∇ θ . matrices Gauss-Newton Precon-ditioning for Natural Gradient (Amari, 1998) / Generalized Gauss-Newton (Schraudolph, 2001) / TONGA (Le Roux et al., 2008) can all be expressed as either (centered) covariance or (un-centered) second…”
Approximate Fisher Information Matrix to Characterize the Training of Deep Neural Networks (2018, methodology): “The (Generalized) Gauss-Newton method [19], [20] approximates Hessian with the Gauss-Newton matrix.”