Quasi-Newton Methods: A New Direction

By Philipp Hennig, Martin Kiefel. 2012

110 citations recorded by the source corpus.

Four decades after their invention, quasi-Newton methods are still state of the art in unconstrained numerical optimization. Although not usually interpreted thus, these are learning algorithms that fit a local quadratic approximation to the objective function. We show that many, including the most popular, quasi-Newton methods can be interpreted as approximations of Bayesian linear regression under varying prior assumptions. This new notion elucidates some shortcomings of classical algorithms, and lights the way to a novel nonparametric quasi-Newton method, which is able to make more efficient use of available information at computational cost similar to its predecessors.

Subfields: Computational Theory and Mathematics, Numerical Analysis

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Theory and Mathematics

1. Michael A. Osborne
2. David Kristjanson Duvenaud
3. Michael Schober
4. Roman Garnett
5. Philipp Hennig
6. Nando de Freitas
7. Ziyun Wang

In Numerical Analysis

1. David Kristjanson Duvenaud
2. Michael Schober
3. Michael A. Osborne
4. Roman Garnett
5. Philipp Hennig

Citations in the corpus

• Fast Probabilistic Optimization from Noisy Gradients (2013, methodology): “Analogously to the L-BFGS method (Nocedal, 1980), we simply impose a memory limit Mmax and ignore all observations more than Mmax steps away (see also Hennig & Kiefel, 2012).”
• Active Learning of Linear Embeddings for Gaussian Processes (2013, background): “…s) (Rasmussen & Williams, 2006) has become popular for regression, clas-sification, quadrature (O’Hagan, 1991), and global optimization (Brochu et al., 2010), such approaches remain intractable for large numbers of input variables (with the exception of local optimization (Hennig & Kiefel, 2012)).”
• Probabilistic ODE Solvers with Runge-Kutta Means (2014, methodology): “To keep notation simple, we will treat x as scalar-valued; the multivariate extension is straightforward (it involves N separate GP models, explained in supp.).”