Training Neural Networks with Stochastic Hessian-Free Optimization
By Ryan Kiros. 2013
52 citations recorded by the source corpus.
Hessian-free (HF) optimization has been successfully used for training deep autoencoders and recurrent networks. HF uses the conjugate gradient algorithm to construct update directions through curvature-vector products that can be computed on the same order of time as gradients. In this paper we exploit this property and study stochastic HF with gradient and curvature mini-batches independent of the dataset size. We modify Martens' HF for these settings and integrate dropout, a method for preventing co-adaptation of feature detectors, to guard against overfitting. Stochastic Hessian-free optimization gives an intermediary between SGD and HF that achieves competitive performance on both classification and deep autoencoder experiments.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Quoc V. Le
- Mohit Bansal
- Sergey Levine
- Ruslan Salakhutdinov
- Roger Baker Grosse
- Kyunghyun Cho
- Jianfeng Gao
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Jitendra Malik
- Ming-Hsuan Yang
- Bernt Schiele
- Xiaogang Wang
- Andrew Zisserman
In Optimization & Training
- Yoshua Bengio
- James Martens
- Razvan Pascanu
- Roger Baker Grosse
- Stefano Soatto
- Ilya Sutskever
- Quoc V. Le
- Aaron C. Courville
Citations in the corpus
- Revisiting Natural Gradient for Deep Networks (2013, positive): “Our result is in the spirit of the work of Kiros (2013), though the exact approach of dealing with small minibatches is slightly different.”
- Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs) (2015, background): “Small means may be achieved by (i) unit zero mean normalization LeCun et al. (1991; 1998); Schraudolph (1998); Cho et al.”
- Optimizing Neural Networks with Kronecker-factored Approximate Curvature (2015, methodology): “A version of momentum is also present in the original HF method, and it plays an arguably even more important role in more “stochastic” versions of HF (Martens and Sutskever, 2012; Kiros, 2013).”
- Combining stochastic average gradient and Hessian-free optimization for sequence training of deep neural networks (2013, methodology): “Also, [6] integrates dropout which we do not address in our paper.”
- A Trace-restricted Kronecker-Factored Approximation to Natural Gradient (2020, unknown): “…Keskar and Berahas, 2016; Berahas et al., 2019; Gold-farb et al., 2020), Hessian-Free optimization approach (Martens, 2010; Martens and Ilya, 2011; Kiros, 2013; Pan et al., 2017), Kronecker-factored Approximate Curvature (KFAC) (Martens and Grosse, 2015; Grosse and Martens, 2016; Martens et al.,…”
- Approximate Fisher Information Matrix to Characterize the Training of Deep Neural Networks (2018, methodology): “This Hessian-free method has been successfully applied to train neural networks [27], [28].”
- Training Deep and Recurrent Networks with Hessian-Free Optimization (2012, methodology): “Recent work by Martens and Sutskever (2011) has demonstrated that HF is a viable method for optimizing RNNs on datasets that exhibit pathological long range dependencies that were believed difficult or impossible to learn with gradient descent.”