On the Acceleration of L-BFGS with Second-Order Information and Stochastic Batches

By Jie Liu, Yu Rong, Martin Takác, Junzhou Huang. 2018

7 citations recorded by the source corpus.

This paper proposes a framework of L-BFGS based on the (approximate) second-order information with stochastic batches, as a novel approach to the finite-sum minimization problems. Different from the classical L-BFGS where stochastic batches lead to instability, we use a smooth estimate for the evaluations of the gradient differences while achieving acceleration by well-scaling the initial Hessians. We provide theoretical analyses for both convex and nonconvex cases. In addition, we demonstrate that within the popular applications of least-square and cross-entropy losses, the algorithm admits a simple implementation in the distributed environment. Numerical experiments support the efficiency of our algorithms.

Subfields: Artificial Intelligence, Computational Mechanics, Statistical and Nonlinear Physics, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Statistical and Nonlinear Physics

In Optimization & Training

Citations in the corpus

Fast and Furious Convergence: Stochastic Second Order Methods under Interpolation (2019, methodology): “Next, we study stochastic Quasi-Newton meth-ods (Schraudolph et al., 2007; Sunehag et al., 2009; Zhou et al., 2017; Liu et al., 2018; Berahas et al., 2019; Gao and Goldfarb, 2019) in the interpolation setting (Section 5).”
Quasi-Newton Methods for Deep Learning: Forget the Past, Just Sample (2019, methodology): “Our methods borrow some of the ideas proposed in [10, 23, 37].”