Quasi-Newton Methods, Motivation and Theory

By John E. Dennis Jr., Jorge J. Moré. 1974

This paper is an attempt to motivate and justify quasi-Newton methods as useful modifications of Newton''s method for general and gradient nonlinear systems of equations. References are given to ample numerical justification; here we give an overview of many of the important theoretical results and each is accompanied by sufficient discussion to make the results and hence the methods plausible. Key Words and Phrases: unconstrained minimization, nonlinear simultaneous equations, update methods, quasi-Newton methods.

Subfields: Mathematical Optimization, Numerical Methods & Scientific Computing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Mathematical Optimization

1. Jorge Nocedal
2. Philippe L. Toint
3. A. Petersen‐Øverleir
4. Jascha Narain Sohl-Dickstein
5. Eric P. Xing
6. N. Gould
7. Hosagrahar Visvesvaraya Jagadish
8. Masao Fukushima

In Numerical Methods & Scientific Computing

1. Maruan Al-Shedivat
2. A. Petersen‐Øverleir
3. Jascha Narain Sohl-Dickstein
4. Eric P. Xing
5. N. Gould
6. Hosagrahar Visvesvaraya Jagadish
7. Philippe L. Toint
8. Masao Fukushima

Citations in the corpus

• Practical tradeoffs between memory, compute, and performance in learned optimizers (2022, methodology): “…of approximations – e.g. diagonal approximations (Duchi et al., 2011), block diagonal (Martens and Grosse, 2015; Gupta et al., 2018), as well as a large family of quasi newton methods (Dennis and Moré, 1977) (e.g. BFGS (Broyden, 1970) and its low memory counterpart, L-BFGS (Liu and Nocedal, 1989)).”
• A descent modified Polak–Ribière–Polyak conjugate gradient method and its global convergence (2006, methodology): “In order to introduce our method, let us simply recall the well-known BFGS quasi-Newton method (see, e.g. Dennis & Moré, 1974, 1977).”
• Machine Learning-augmented Predictive Modeling of Turbulent Separated Flows over Airfoils (2016, methodology): “5 uses a gradient-based Quasi-Newton method employing the limited memory Broyden- Fletcher-Goldfarb-Shanno algorithm [54].”
• Fast Probabilistic Optimization from Noisy Gradients (2013, background): “…reach the minimum in one step, and in the sense that there is a neighbourhood around each local minimum x∗ such that, if the algorithm reaches this neighbourhood, the estimate xt converges to x ∗ quadratically fast (Dennis & Morée, 1977) – although this neighbourhood can be arbitrarily small.”
• A modified BFGS method and its global convergence in nonconvex minimization (2001, methodology): “Let {xk} be generated by MBFGS method with Bk being updated by (2:12).”
• Evaluation of Optimization Methods for Nonrigid Medical Image Registration Using Mutual Information and B-Splines (2007, methodology): “Given certain conditions, many QN methods can be shown to be superlinearly convergent [ 14 ]”
• Convergence of quasi-Newton matrices generated by the symmetric rank one update (1991, background): “We refer the reader to [3, 4, 6, 8] for further motivation and analysis concerning these now classical algorithms.”