Deep learning via Hessian-free optimization

By James Martens. 2010

1,146 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

Exact solutions to the nonlinear dynamics of learning in deep linear neural networks (2013, neutral): “At the same time, recent results have obtained excellent performance starting from carefullyscaled random initializations, though interestingly, pretrained initializations still exhibit faster convergence [21, 13, 22, 3, 4, 1, 23] (see Supplementary Appendix D for discussion).”
Pushing Stochastic Gradient towards Second-Order Methods -- Backpropagation Learning with Transformations in Nonlinearities (2013, background): “Some later works have returned to traditional back-propagation learning in deep models and noticed that it can also provide impressive results [6] given either a sophisticated learning algorithm [9] or simply enough computational power [2].”
Fast large-scale optimization by unifying stochastic gradient and quasi-Newton methods (2013, methodology): “For Hessian-free , the hyperparameters, source code, and objec-tive function are identical to those used in (Martens, 2010), and the training data was divided into four “chunks.””
Revisiting Natural Gradient for Deep Networks (2013, methodology): “HessianFree optimization (Martens, 2010; Sutskever et al., 2011; Chapelle and Erhan, 2011), Krylov Subspace Descent (Vinyals and Povey, 2012), natural gradient descent (Amari, 1997; Park et al.”
Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks (2016, background): “If the condition number of the Hessian matrix of the objective at the optimum is low, the problem is said to exhibit pathological curvature , and ﬁrst-order gradient descent will have trouble making progress [18, 28].”
Gradient Estimation Using Stochastic Computation Graphs (2015, methodology): “A variant of this technique, called Hessian-free optimization [13], has been used to train large neural networks.”
Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs) (2015, methodology): “The recently introduced Hessian-Free Optimization technique (Martens, 2010) and the Krylov Subspace Descent methods (Vinyals & Povey, 2012) use an extended Gauss-Newton approximation of the Hessian, therefore they can be interpreted as versions of natural gradient descent (Pascanu & Bengio, 2014).”
Learning to Execute (2014, methodology): “…to use Recurrent Neural Networks with Long Short Term Memory units (Hochreiter & Schmidhuber, 1997), although there are many other RNN variants that perform well on tasks with long term dependencies (Cho et al., 2014; Jaeger et al., 2007; Koutn´ık et al., 2014; Martens, 2010; Bengio et al., 2013).”
Multiset-Equivariant Set Prediction with Approximate Implicit Differentiation (2021, background): “∇ This can be problematic when H is ill-conditioned or singular, so we regularize it with the identity matrix through H (cid:48) = H /γ + I with γ > 0 (Martens, 2010; Rajeswaran et al., 2019).”
Fast and Furious Convergence: Stochastic Second Order Methods under Interpolation (2019, methodology): “…in explicitly incorporating second-order information through the Hessian matrix in Newton methods (Liu and Nocedal, 1989; Nocedal and Wright, 2006; Martens, 2010) or through the Fisher information matrix in natural gradient meth-ods (Amari, 1998; Roux et al., 2008; Pascanu and Ben-gio, 2014;…”
Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data (2018, background): “We also note that it is essential to carefully set the scale of the initialization, which is a extensively studied topic [20, 28].”
Representation Learning: A Review and New Perspectives (2012, methodology): “This optimization difficulty has motivated the exploration of second-order methods for deep architectures and recurrent networks, in particular Hessianfree second-order methods [142], [143].”
Identifying and attacking the saddle point problem in high-dimensional non-convex optimization (2014, neutral): “69 achieved by the Hessian-Free method (Martens, 2010).”
How to Construct Deep Recurrent Neural Networks (2013, methodology): “We ﬁnd some of the recently introduced approaches, such as advanced regularization methods (Pascanu et al. , 2013a) and advanced optimization algorithms (see, e.g., Pascanu and Bengio, 2013; Martens, 2010), to be promising candidates.”
Optimizing Neural Networks with Kronecker-factored Approximate Curvature (2015, methodology): “The GGN has served as the curvature matrix of choice in Hessian-free optimization (HF) (Martens, 2010) and re-lated methods, and so in light of its equivalence to the Fisher, these 2nd-order methods can be seen as approximate natural gradient methods.”
Empirical Analysis of the Hessian of Over-Parametrized Neural Networks (2017, background): “ecision. 2.2 GENERALIZED GAUSS-NEWTON DECOMPOSITION OF THE HESSIAN In order to study its spectrum, we will describe how the Hessian can be decomposed into two meaningful matrices (LeCun et al., 1998; Martens, 2010; 2014). Suppose the loss function is given as a composition of two functions, the mode”
Understanding Black-box Predictions via Influence Functions (2017, methodology): “While an exact solution takes p CG iterations, in practice we can get a good approximation with fewer iterations; see Martens (2010) for more details.”
Minimizing finite sums with the stochastic average gradient (2013, background): “There has been a substantial amount ofwork ondeveloping stochastic variants of these algorithms, with several of the notable recent examples including [6,18,22,36,60,63].”
Stronger data poisoning attacks break data sanitization defenses (2018, methodology): “ loss max(0;1 y>x). To eciently compute @^ @~x in (10), we follow Koh and Liang (2017) and use a combination of fast Hessian-vector products (Pearlmutter, 1994) and a conjugate gradient solver (Martens, 2010); see also Agarwal et al. (2016) and Munoz-Gonz~ alez et al. (2017) for other tractable a”
Neural Optimizer Search with Reinforcement Learning (2017, methodology): “…to train, and many methods have been designed to reduce this difﬁculty (e.g., Riedmiller & Braun (1992); LeCun et al. (1998); Schraudolph (2002); Martens (2010); Le et al. (2011); Duchi et al. (2011); Zeiler (2012); Martens & Sutskever (2012); Schaul et al. (2013); Pascanu & Bengio (2013);…”