Quasi-Newton methods: superlinear convergence without line searches for self-concordant functions

By Wenbo Gao, Donald Goldfarb. 2016

54 citations recorded by the source corpus.

ABSTRACT We consider the use of a curvature-adaptive step size in gradient-based iterative methods, including quasi-Newton methods, for minimizing self-concordant functions, extending an approach first proposed for Newton's method by Nesterov. This step size has a simple expression that can be computed analytically; hence, line searches are not needed. We show that using this step size in the BFGS method (and quasi-Newton methods in the Broyden convex class other than the DFP method) results in superlinear convergence for strongly convex self-concordant functions. We present numerical experiments comparing gradient descent and BFGS methods using the curvature-adaptive step size to traditional methods on deterministic logistic regression problems, and to versions of stochastic gradient descent on stochastic optimization problems.

Subfields: Artificial Intelligence, Computational Mechanics, Numerical Analysis, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Numerical Analysis

In Optimization & Training

Citations in the corpus

Fast and Furious Convergence: Stochastic Second Order Methods under Interpolation (2019, methodology): “Next, we study stochastic Quasi-Newton meth-ods (Schraudolph et al., 2007; Sunehag et al., 2009; Zhou et al., 2017; Liu et al., 2018; Berahas et al., 2019; Gao and Goldfarb, 2019) in the interpolation setting (Section 5).”