On the Use of Stochastic Hessian Information in Optimization Methods for Machine Learning

By Richard H. Byrd, Gillian M. Chin, Will Neveitt, Jorge Nocedal. 2011

288 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computational Mechanics, Numerical Analysis, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Numerical Analysis

In Optimization & Training

Citations in the corpus

Fast large-scale optimization by unifying stochastic gradient and quasi-Newton methods (2013, background): “In (Martens, 2010), (Byrd et al., 2011), and (Vinyals & Povey, 2011) stochastic versions of Hessian-free optimization are implemented and applied to optimization of deep networks.”
Fast and Furious Convergence: Stochastic Second Order Methods under Interpolation (2019, methodology): “Besides the works discussed earlier, there have been many recent works analyzing subsampled Newton meth-ods (Byrd et al., 2011, 2012; Milzarek et al., 2019; Li et al., 2019).”
Optimization Methods for Large-Scale Machine Learning (2016, background): “hen S k f1;:::;ngfor all k2N. Specically, due to the observations above, if the subsampled Hessian is positive denite, then any number of CG iterations yields a direction of sucient descent for R n[26]. Although it has not been established that the rate of convergence for such a method is faster tha”
Hybrid Deterministic-Stochastic Methods for Data Fitting (2011, background): “Another approach, not considered here, is to base the quasi-Newton Hessian on an independent set of sampled gradients [7].”
An efficient distributed learning algorithm based on effective local functional approximations (2013, methodology): “Literature shows that quadratic approximations have good robustness properties; for example, subsampling in Hessian computation (Byrd et al., 2012) doesn’t worsen direction ﬁnding much.”
PyHessian: Neural Networks Through the Lens of the Hessian (2019, background): “[10] Richard H Byrd, Gillian M Chin, Will Neveitt, and Jorge Nocedal.”
Quasi-Newton Methods for Deep Learning: Forget the Past, Just Sample (2019, methodology): “In order to alleviate these issues, stochastic Newton [4, 9, 39, 49, 55] and stochastic quasi-Newton [2, 10, 15, 23, 27, 42, 51] methods have been proposed.”
Do Subsampled Newton Methods Work for High-Dimensional Data? (2019, methodology): “Recent works proposed the Sub-Sampled Newton (SSN) methods for reducing the per-iteration complexity of the Newton’s method (Byrd et al. 2011; Pilanci and Wain-wright 2015; Roosta-Khorasani and Mahoney 2016; Pi-lanci and Wainwright 2017; Xu, Roosta Khorasan, and Ma-honey 2017; Berahas,…”
Sub-Sampled Newton Methods I: Globally Convergent Algorithms (2016, methodology): “Our algorithms are designed for the following settings.”
Second Order Stochastic Optimization in Linear Time (2016, unknown): “The line of work which is closest to ours employs subsampling to create estimates of the true Hessian [Mar10, BCNN11].”
Accelerating SVRG via second-order information (2015, methodology): “Naturally, there have also been quite a few recent works that aim to accelerate convergence of stochastic methods by incorporating curvature information [3, 4, 16, 10, 11], while maintaining a per-iteration complexity of O(p).”
Newton-Stein Method: An Optimization Method for GLMs via Stein's Lemma (2015, methodology): “Another natural estimator would be the sub-sampled Hessian method suggested by [Mar10, BCNN11].”
Least Squares Revisited: Scalable Approaches for Multi-class Prediction (2013, methodology): “There has also been a renewed interest in Quasi-Newton methods scalable to statistical problems using stochastic approximation ideas (Byrd et al., 2011; Bordes et al., 2009).”