A Stochastic Quasi-Newton Method for Online Convex Optimization

By N. Schraudolph, Jin Yu, Simon Günter. 2007

427 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

Fast and Furious Convergence: Stochastic Second Order Methods under Interpolation (2019, methodology): “Next, we study stochastic Quasi-Newton meth-ods (Schraudolph et al., 2007; Sunehag et al., 2009; Zhou et al., 2017; Liu et al., 2018; Berahas et al., 2019; Gao and Goldfarb, 2019) in the interpolation setting (Section 5).”
Optimization Methods for Large-Scale Machine Learning (2016, methodology): “s be avoided in the stochastic regime? There have been various proposals to avoid dierencing noisy gradient estimates. One possibility is to employ the same sample when computing gradient dierences [18,142]. An alternative approach that allows greater freedom in the choice of the stochastic gradient”
Bayesian Posterior Sampling via Stochastic Gradient Fisher Scoring (2012, neutral): “…perspective one may view this algo-rithm as a Fisher scoring method based on stochastic gradients (see e.g. (Schraudolph et al., 2007)) but in such a way that the randomness introduced in the subsampling process is used to sample from the posterior distribution when we arrive at its mode.”
Variable Metric Stochastic Approximation Theory (2009, methodology): “Since recent online quasi-Newton methods (Schraudolph et al., 2007) do not provide convergence of their scaling matrices, we now employ the RobbinsSiegmund theorem to prove that such updates are still guaranteed almost sure convergence, provided the spectrum of their (possibly non-convergent)…”
Global convergence of online limited memory BFGS (2014, methodology): “This has resulted in the development of the stochastic quasi-Newton methods known as online (o) Broyden-Fletcher-Goldfarb-Shanno (BFGS) (Schraudolph et al. (2007)), regularized stochastic BFGS (RES) (Mokhtari and Ribeiro (2014a)), and online limited memory (oL)BFGS (Schraudolph et al. (2007)) which…”
Recent Advances of Large-Scale Linear Classification (2012, methodology): “Some available optimization methods include L-BFGS (quasi Newton) and conjugate gradient [114], stochastic gradient descent [115], stochastic quasi Newton [96], [116], and trust region Newton method [117].”
Adaptive Newton Method for Empirical Risk Minimization to Statistical Accuracy (2016, methodology): “They do improve convergence constants in ill-conditioned problems but they still converge at linear rates.”
Metaheuristic design of feedforward neural networks: A review of two decades of research (2017, methodology): “Quasi-Newton uses a second-order partial derivative (7) of error (4), and it computes its weight search direction by using Broyden-Fletcher-Goldfarb-Shanno (BFGS) method [78].”
IQN: An Incremental Quasi-Newton Method with Local Superlinear Convergence Rate (2017, methodology): “Theﬁrstattempt to replace stochastic gradients for gradients in the update of quasi-Newton methods was the work in [29] which introduces a stochastic (online) version of the BFGS quasi-Newton method as well as its limited memory variant.”
A Variance Reduced Stochastic Newton Method (2015, methodology): “A stochastic version of BFGS known as oBFGS was proposed in (Schraudolph et al., 2007) in which stochastic gradients are used for both the determination of the descent direction and the approximation of the inverse Hessian.”
Deep Convolutional Neural Networks Versus Multilayer Perceptron for Financial Prediction (2018, methodology): “We propose to perform MLP training with BroydenFletcher-Goldfarb-Shannon (BFGS) quasi-Newton algorithm [16], that has proved to lead to significantly better performances than those obtained by classical Levenberg-Marquardt backpropagation [11].”
Asynchronous Stochastic Quasi-Newton MCMC for Non-Convex Optimization (2018, methodology): “As a remedy, stochastic L-BGFS methods have been proposed (Byrd et al., 2016; Schraudolph et al., 2007; Moritz et al., 2016; Zhou et al., 2017; Yousefian et al., 2017; Zhao et al., 2017), which aim to reduce the computational requirements of L-BFGS by replacing ∇U (i.”
Fast Stochastic Frank-Wolfe Algorithms for Nonlinear SVMs (2010, neutral): “The proposed algorithms achieve comparable or even better accuracies than the state-of-the- art methods, and are significantly faster.”
A Linearly-Convergent Stochastic L-BFGS Algorithm (2015, background): “In particular, much progress has gone toward designing stochastic variants of L-BFGS (Mokhtari and Ribeiro, 2014a; Wang et al., 2014; Byrd et al., 2014; Bordes et al., 2009; Schraudolph et al., 2007; Sohl-Dickstein et al., 2014).”
Second Order Stochastic Optimization in Linear Time (2016, methodology): “Other methods include taking a hybrid approach that combine the above two methods to gain improvement [SYG07, BHNS14].”