A fast natural Newton method

By Nicolas Le Roux, Andrew William Fitzgibbon. 2010

62 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

Adam: A Method for Stochastic Optimization (2014, methodology): “Other stochastic optimization methods include vSGD (Schaul et al., 2012), AdaDelta (Zeiler, 2012) and the natural Newton method from Roux & Fitzgibbon (2010), all setting stepsizes by estimating curvature from ﬁrst-order information.”
Revisiting Natural Gradient for Deep Networks (2013, methodology): “Note that in Roux and Fitzgibbon (2010) a method for combining second order information and TONGA is proposed.”
New Insights and Perspectives on the Natural Gradient Method (2014, methodology): “And second, as we establish in Section 9, it is very often equivalent to the so-called Generalized Gauss-Newton matrix (GGN) (Schraudolph, 2002; Martens and Sutskever, 2012) discussed in Section 8, which is a well established and rigorously justified approximation of the Hessian that has been used i”
Probabilistic Line Searches for Stochastic Optimization (2015, methodology): “…et al., 2011), ‘stochastic meta-descent’ (Schraudolph, 1999), empirical estimates for the natural gradient (Amari et al., 2000) or the Newton direction (Roux & Fitzgibbon, 2010), to problem-speciﬁc algorithms for speciﬁc optimization tasks (Rajesh et al., 2013), and more elaborate estimates of the…”
Newton-Stein Method: An Optimization Method for GLMs via Stein's Lemma (2015, methodology): “Well-known composite algorithms are the combinations of sub-sampling and Quasi-Newton [SYG07, BHNS14], SGD and GD [FS12], NG and NM [LRF10], NG and low-rank approximation [LRMB08], sub-sampling and eigenvalue thresholding [EM15].”
Adaptive learning rates and parallelization for stochastic, sparse, non-smooth gradients (2013, background): “mplement and of linear complexity. 2 Background There are a number of adaptive settings for SGD learning rates, or equivalently, diagonal preconditioning schemes, to be found in the literature, e.g., [2, 3, 4, 5, 6, 7]. The aim of those is generally to increase performance on stochastic optimization”
Stochastic Quasi-Newton Methods for Nonconvex Stochastic Optimization (2016, background): “Roux and Fitzgibbon [46] discussed the necessity of including both Hessian and covariance matrix information in a stochastic Newton type method.”
A modified Adam algorithm for deep neural network optimization (2023, methodology): “Other stochastic optimization methods discussed include vSGD [21], AdaDelta [22], and the natural newton method [23].”
Limitations of the Empirical Fisher Approximation (2019, background): “Le Roux et al. [2007] and Le Roux and Fitzgibbon [2010] have considered the empirical Fisher in its interpretation as the (non-central) covariance matrix of stochastic gradients.”
A Stochastic Quasi-Newton Method for Large-Scale Optimization (2014, methodology): “l, the parameters L; M and b H provide much freedom in adapting the SQN method to a specic application. 4 Related Work Various stochastic quasi-Newton algorithms have been proposed in the literature [23, 13, 4, 21], but have not been entirely successful. The methods in [23] and [13] use the BFGS fra”
Convergence rates of sub-sampled Newton methods (2015, background): “Examples include, sub-sampling and Quasi-Newton [BHNS14], SGD and GD [FS12], NGD and NM [LRF10], NGD and low-rank approximation [LRMB08].”
Policy evaluation with temporal differences: a survey and comparison (2015, methodology): “other and the mean squared error have been examined thoroughly by Schoknecht (2002) and Scherrer (2010) and in parts by Bach and Moulines (2011), Lazaric et al.”