SGD-QN: Careful Quasi-Newton Stochastic Gradient Descent

By Antoine Bordes, L. Eon Bottou, Patrick Gallinari. 2009

403 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

Fast large-scale optimization by unifying stochastic gradient and quasi-Newton methods (2013, methodology): “Stochastic meta-descent (Schraudolph, 1999), AdaGrad (Duchi et al., 2010), and SGD-QN (Bordes et al., 2009) rescale the gradient independently for each dimension, and can be viewed as accumulating something similar to a diagonal approximation to the Hessian.”
No more pesky learning rates (2012, methodology): “…amounts of streaming data, to theoretical optimality results for generalization error (Bottou & LeCun, 2004), and to competitions being won by SGD methods, such as the PAS-CAL Large Scale Learning Challenge (Bordes et al., 2009), where Quasi-Newton approximation of the Hessian was used within SGD.”
SGDR: Stochastic Gradient Descent with Warm Restarts (2016, background): “Despite some recent progress in understanding and addressing the latter problems (Bordes et al., 2009; Dauphin et al., 2014; Choromanska et al., 2014; Dauphin et al., 2015), state-of-the-art optimization techniques attempt to approximate the inverse Hessian in a reduced way, e.g., by considering…”
Optimization Methods for Large-Scale Machine Learning (2016, methodology): “s be avoided in the stochastic regime? There have been various proposals to avoid dierencing noisy gradient estimates. One possibility is to employ the same sample when computing gradient dierences [18,142]. An alternative approach that allows greater freedom in the choice of the stochastic gradient”
Minimizing finite sums with the stochastic average gradient (2013, background): “There has been a substantial amount ofwork ondeveloping stochastic variants of these algorithms, with several of the notable recent examples including [6,18,22,36,60,63].”
Federated Optimization: Distributed Machine Learning for On-Device Intelligence (2016, methodology): “A third class of new algorithms are the Stochastic quasi-Newton methods [16, 9].”
Layerwise Optimization by Gradient Decomposition for Continual Learning (2021, neutral): “Stochastic Gradient Descent is the most widely used optimization techniques for training DNNs [3, 31, 2].”
New Insights and Perspectives on the Natural Gradient Method (2014, methodology): “And while Bordes et al. (2009) presents their bounds with F̄ in place of Σg, these are the same matrix when evaluated at θ = θ∗ as we have E[g(θ∗)] = 0 (since θ∗ is a local optimum).”
Practical Recommendations for Gradient-Based Training of Deep Architectures (2012, background): “…many authors have advocated exploring adaptive learning rate methods, either for a global learning rate (Cho et al. , 2011), a layer-wise learning rate, a neuron-wise learning rate, or a parameter-wise learning rate (Bordes et al. , 2009) (which then starts to look like a diagonal Newton method).”
Distributed optimization with arbitrary local solvers (2015, background): “an operation one generally tries to avoid. This and several other practical issues have been recently addressed in [2]. Finally, another class of extensions to SGD are stochastic quasi-Newton methods [6, 11]. Despite their clear potential, a lack of theoretical understanding and complicated implemen”
Online Batch Selection for Faster Training of Neural Networks (2015, background): “Despite some recent progress in understanding and addressing the latter problems (Bordes et al., 2009; Dauphin et al., 2014; Choromanska et al., 2014; Dauphin et al., 2015), state-of-the-art optimization techniques tend to approximate the inverse Hessian in a reduced way, e.g., by considering only…”
Stochastic, Distributed and Federated Optimization for Machine Learning (2017, methodology): “Finally, another class of extensions to SGD are stochastic quasi-Newton methods [17, 26].”
SDNA: Stochastic Dual Newton Ascent for Empirical Risk Minimization (2015, background): “An exception in this regard is the work of Bordes et al. (2009), who give a O(1/ ) complex- ar X iv :1 50 2.”
A Stochastic Quasi-Newton Method for Large-Scale Optimization (2014, methodology): “[4] updates the rescaling matrix at fixed intervals in order to reduce computational costs.”
Global convergence of online limited memory BFGS (2014, background): “In oBFGS this possibility precludes convergence analyses (Bordes et al. (2009); Schraudolph et al. (2007)) and may result in erratic numerical behavior (Mokhtari and Ribeiro (2014a)).”
A Survey of Optimization Methods From a Machine Learning Perspective (2019, background): “Then, the relevant experts and scholars also introduced this stochastic idea to the second-order optimization methods [149]–[151] and achieved good results.”
Towards Optimal One Pass Large Scale Learning with Averaged Stochastic Gradient Descent (2011, methodology): “Hence various methods for approximating H have been proposed (LeCun et al., 1998; Schraudolph et al., 2007; Roux et al., 2008; Bordes et al., 2009).”