Learning Recurrent Neural Networks with Hessian-Free Optimization

By James Martens, Ilya Sutskever. 2011

658 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling (2014, methodology): “…approach is to devise a better learning algorithm than a simple stochastic gradient descent [see, e.g., Bengio et al., 2013, Pascanu et al., 2013, Martens and Sutskever, 2011], for example using the very simple clipped gradient , by which the norm of the gradient vector is clipped, or using…”
Revisiting Natural Gradient for Deep Networks (2013, background): “Structural damping (Martens and Sutskever, 2011), a specific regularization term used to improve training of recurrent neural network, can also be explained from the natural gradient descent perspective.”
Theano: new features and speed improvements (2012, methodology): “A more careful implementation however reveals that two passes should be sufﬁcient (see Martens and Sutskever (2011)).”
On the difficulty of training recurrent neural networks (2012, methodology): “…the multiplication problem, the 3-bit temporal order problem , the random permutation problem and the noise-less memorization problem in two variants (when the pattern needed to be memorized is 5 bits in length and when it contains over 20 bits of information; see Martens and Sutskever (2011)).”
Representation Learning: A Review and New Perspectives (2012, methodology): “This optimization difficulty has motivated the exploration of second-order methods for deep architectures and recurrent networks, in particular Hessianfree second-order methods [142], [143].”
How to Construct Deep Recurrent Neural Networks (2013, methodology): “41 using an RNN trained with the Hessian-free learning algorithm (Martens and Sutskever, 2011).”
Optimization Methods for Large-Scale Machine Learning (2016, methodology): “This approach has been applied successfully in the training of deep neural networks [10, 99].”
Recurrent Batch Normalization (2016, background): “Effective optimization of recurrent neural networks is an active area of study [22, 18, 20].”
Modeling Temporal Dependencies in High-Dimensional Sequences: Application to Polyphonic Music Generation and Transcription (2012, methodology): “It was recently shown that training RNNs via Hessian-free (HF) optimization could help reduce these diﬃculties (Martens & Sutskever, 2011).”
Deep learning in neural networks: An overview (2014, methodology): “2011: Hessian-free optimization for RNNs Also in 2011 it was shown (Martens & Sutskever, 2011) that Hessian-free optimization (e.g., Møller, 1993; Pearlmutter, 1994; Schraudolph, 2002) (Section 5.6.2) can alleviate the Fundamental Deep Learning Problem (Section 5.9) in RNNs, outperforming standard…”
An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling (2018, methodology): “First introduced by Hochreiter & Schmidhuber (1997), the adding problem has been used repeatedly as a stress test for sequence models (Martens & Sutskever, 2011; Pascanu et al., 2013; Le et al., 2015; Arjovsky et al., 2016; Zhang et al., 2016).”
Deep Learning: Methods and Applications (2014, background): “Recent published work showed that there is vast room to improve the current optimization techniques for learning deep architectures [69, 208, 238, 239, 311, 356, 393].”
A Simple Way to Initialize Recurrent Networks of Rectified Linear Units (2015, methodology): “The task is to report the sum of the two real values that are marked by having a 1 as the second input [16, 15, 24].”
New Insights and Perspectives on the Natural Gradient Method (2014, methodology): “As pointed out in Martens and Sutskever (2011), the GGN can also be derived by a generalization of the previously described derivation of the classical Gauss-Newton matrix to the situation where L is an arbitrary convex loss.”
Jointly Modeling Embedding and Translation to Bridge Video and Language (2015, background): “Although the gradients of the RNN are easy to compute, RNN models are difﬁcult to train, especially when the problems have long-range temporal dependencies, due to the well-known “vanishing gradi-ent” effect [4, 21].”
Unitary Evolution Recurrent Neural Networks (2015, methodology): “…as benchmarks for testing the ability of a model to capture long-term memory (Hochreiter & Schmidhuber, 1997; Le et al., 2015; Graves et al., 2014; Martens & Sutskever, 2011) Of the handful of optimization algorithms we tried on the various models, RMSProp (Tieleman & Hinton, 2012) lead to…”