Training Deep and Recurrent Networks with Hessian-Free Optimization

By James Martens, Ilya Sutskever. 2012

259 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Optimization & Training

Citations in the corpus

Training Very Deep Networks (2015, background): “To deal with the difﬁculties of training deep networks, some researchers have focused on developing better optimizers (e.g. [13, 14, 15]).”
Trust Region Policy Optimization (2015, methodology): “For additional exposition on the use of Hessian-vector products for optimizing neural network objectives, see (Martens & Sutskever, 2012) and (Pascanu & Bengio, 2013).”
VirTex: Learning Visual Representations from Textual Annotations (2020, background): “We compensate this by reducing memory usage through gradient checkpointing [102, 103].”
NVAE: A Deep Hierarchical Variational Autoencoder (2020, background): “This trick is known as gradient check-pointing [54, 55] and it requires recomputing BN in the backward pass.”
The Reversible Residual Network: Backpropagation Without Storing Activations (2017, methodology): “The method of Martens and Sutskever [22] requres 2 √ L storage, and it demands an additional forward computation for each layer, leading to a total computational cost of 3L.”
Optimizing Neural Networks with Kronecker-factored Approximate Curvature (2015, background): “…be shown (Martens, 2014; Pas-canu and Bengio, 2014) that the Fisher is equivalent to the Generalized Gauss-Newton matrix (GGN) (Schraudolph, 2002; Martens and Sutskever, 2012) in certain important cases, which is a well-known positive semi-deﬁnite approximation to the Hessian of the objective…”
Gradient-based Hyperparameter Optimization through Reversible Learning (2015, background): “Reversible elementary computation Recurrent neural network models can require so much memory to differentiate that checkpointing is required simply to compute their elementary gradients (Martens & Sutskever, 2012).”
Don't Use Large Mini-Batches, Use Local SGD (2018, methodology): “The spectrum is computed using power iteration [34, 56] with the relative error of 1e-4.”
Neural Optimizer Search with Reinforcement Learning (2017, neutral): “…this difﬁculty (e.g., Riedmiller & Braun (1992); LeCun et al. (1998); Schraudolph (2002); Martens (2010); Le et al. (2011); Duchi et al. (2011); Zeiler (2012); Martens & Sutskever (2012); Schaul et al. (2013); Pascanu & Bengio (2013); Pascanu et al. (2013); Kingma & Ba (2015); Ba et al. (2017)).”
New Insights and Perspectives on the Natural Gradient Method (2014, methodology): “However, attempts to use such methods in combination with the Hessian have yielded lackluster results for neural network optimization compared to methods based on the GGN (Martens, 2010; Martens and Sutskever, 2012; Vinyals and Povey, 2012).”
In-place Activated BatchNorm for Memory-Optimized Training of DNNs (2017, methodology): “This solution can technically still be considered as a form of checkpointing, but instead of recovering information forwards as in [4, 19], we recover it backwards, thus bearing a similarity to reversible nets [7].”
Deep Sequential Neural Network (2014, background): “Many variants of learning algorithms have been proposed, from complex gradient computations [11], to dropout methods [1], but the baseline learning algorithm still consists in recursively computing the gradient by using the back-propagation algorithm and performing (stochastic) gradient descent.”
SwapAdvisor: Pushing Deep Learning Beyond the GPU Memory Limit via Smart Swapping (2020, background): “As a result, [2, 11, 29] deallocate activation tensors after their last usage in the forward-propagation and later recompute the activation tensors when they are needed.”
A Survey of Optimization Methods From a Machine Learning Perspective (2019, methodology): “The HF method with structural damping can effectively train RNNs [176].”
Stable architectures for deep neural networks (2017, background): “o K 0;:::;N 1 and b 0;:::;N 1. The matrix-vector products with the derivative matrix can be computed without its explicit construction through forward and backward propagation, respectively; see also [36]. However, this requires storing (or recomputing) the values at the output layers; see, for inst”
Scaling up Natural Gradient by Sparsely Factorizing the Inverse Fisher Matrix (2015, background): “While it often works well in practice, it has been observed that deep networks suffer from severe curvature problems, and this observation has motivated work on second-order optimization methods (Martens, 2010; Martens & Sutskever, 2012).”
Recent Advances in Recurrent Neural Networks (2017, background): “It belongs to the broad class of approximate Newton methods that are practical for problems of very high dimensionality, such as the training objectives of large neural networks [28].”
Modeling, Inference and Optimization With Composable Differentiable Procedures (2016, background): “Reversible elementary computation Recurrent neural network models can require so much memory to differentiate that checkpointing is required simply to compute their elementary gradients [66].”