Improved Preconditioner for Hessian Free Optimization

By Olivier Chapelle. 2011

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

Exact solutions to the nonlinear dynamics of learning in deep linear neural networks (2013, neutral): “At the same time, recent results have obtained excellent performance starting from carefullyscaled random initializations, though interestingly, pretrained initializations still exhibit faster convergence [21, 13, 22, 3, 4, 1, 23] (see Supplementary Appendix D for discussion).”
Revisiting Natural Gradient for Deep Networks (2013, methodology): “Hessian-Free optimization (Martens, 2010; Sutskever et al. , 2011; Chapelle and Erhan, 2011), Krylov Sub-space Descent (Vinyals and Povey, 2012), natural gradient descent (Amari, 1997; Park et al. , 2000), TONGA (Le Roux et al. , 2008; Roux and Fitzgibbon, 2010) are just a few of such recently…”
No more pesky learning rates (2012, methodology): “There exist a number of methods for obtaining an on-line estimates of the diagonal Hessian (Martens et al., 2012; Bordes et al., 2009; Chapelle & Erhan, 2011).”
New Insights and Perspectives on the Natural Gradient Method (2014, methodology): “Interestingly however, there are stochastic estimation methods (Chapelle and Erhan, 2011; Martens et al., 2012) which can be used to efficiently estimate the diagonal (or various blocks) of the standard Fisher F , and these work quite well in practice.”
A Kronecker-factored approximate Fisher matrix for convolution layers (2016, background): “It has been observed that a good choice of preconditioner can be crucial to obtaining good performance from iterative methods (Martens, 2010; Chapelle & Erhan, 2011; Vinyals & Povey, 2012).”
Training Deep and Recurrent Networks with Hessian-Free Optimization (2012, methodology): “Chapelle and Erhan [10] give a randomized algorithm for computing an unbiased estimate of the diagonal of the generalized Gauss-Newton matrix, which requires the same amount of work as computing the gradient.”
Ellipsoidal Trust Region Methods and the Marginal Value of Hessian Information for Neural Network Training (2019, methodology): “Many algorithms in the existing literature replace the use of ∇(2)L(wt) in (37) with the Generalized Gauss Newton matrix [52, 19] or an approximation of the latter [54].”
On the importance of initialization and momentum in deep learning (2013, methodology): “Truncated Newton methods, that include the HF method of Martens (2010) as a particular example, work by optimizing a local quadratic model of the objective via the linear conjugate gradient algorithm (CG), which is a ﬁrst-order method.”
Estimating the Hessian by Back-propagating Curvature (2012, methodology): “For example, the diagonal of the Hessian can be used as a preconditioner for first and second order nonlinear optimizers, which is the motivating application of Becker and Le Cun (1988) and Chapelle and Erhan (2011).”
New perspectives on the natural gradient method (2014, background): “…we could work with more directly, we must take non-negligable discrete steps in the given parameter space.3 3In principle, we could move to a much more general class of distributions, such as those given by some nonparametric formulation, where we could work directly with the distributions…”
Training Neural Networks with Stochastic Hessian-Free Optimization (2013, methodology): “Other work on using second-ordermethods for neural networks include [19] who proposed using the Jacobi pre-conditioner for HF, [20] using HF to generate text in recurrent networks and [21] who explored training with Krylov subspace descent (KSD).”
Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training (2023, unknown): “Several previous works (Becker & Le Cun, 1988; Chapelle et al., 2011; Schaul et al., 2013), including the recent work AdaHessian (Yao et al., 2021), use diagonal Hessian as a pre-conditioner in optimizers for training neural networks.”
Adaptive norms for deep learning with regularized Newton methods (2019, methodology): “Furthermore, the preconditioners used in Martens (2010) and Chapelle & Erhan (2011), namely diagonal estimates of the empirical Fisher and Fisher matrix, respectively, can directly be used as matrix norms in our ellipsoidal TR framework.”
Metric-Free Natural Gradient for Joint-Training of Boltzmann Machines (2013, methodology): “Pre-conditioning is also a well-known method for accelerating the convergence speed of linear solvers [5].”