Updating Quasi-Newton Matrices With Limited Storage

By Jorge Nocedal. 1980

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

Learning Dependency-Based Compositional Semantics (2011, methodology): “Nonetheless, with a proper α, SGD converges much faster than L-BFGS and even to a slightly better solution.”
Optimization Methods for Large-Scale Machine Learning (2016, methodology): “trices, even when the exact Hessians are sparse, restricting its use to small and midsize problems. A common solution for this is to employ a limited memory scheme, leading to a method such as L-BFGS [97,113]. A key feature of this approach is that the matrices fH kgneed not be formed explicitly; in”
On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization (2018, methodology): “Adaptive preconditioning with only ﬁrst-order (gradient) information was popularized by the BFGS al-gorithm and its variants ( cf. Nocedal (1980)).”
From amateurs to connoisseurs: modeling the evolution of user expertise through online reviews (2013, methodology): “In practice we optimize model parameters during each iteration us-ing L-BFGS [31], a quasi-Newton method for non-linear optimization of problems with many variables.”
Accessorize to a Crime: Real and Stealthy Attacks on State-of-the-Art Face Recognition (2016, background): “For DNNs, f ( · ) is diﬀerentiable, as this is necessary for training the network via back-propagation.”
Determinantal Point Processes for Machine Learning (2012, methodology): “To optimize the log-likelihood, we will use standard algorithms such as gradient ascent or L-BFGS [113].”
DeltaGrad: Rapid retraining of machine learning models (2020, methodology): “We leverage the L-BFGS algorithm to approximate this, see e.g. Matthies & Strang (1979); Nocedal (1980); Byrd et al. (1994; 1995); Zhu et al. (1997); No-cedal & Wright (2006); Mokhtari & Ribeiro (2015) and references therein.”
Learning for Semantic Parsing with Statistical Machine Translation (2006, positive): “We also implemented limited-memory BFGS (Nocedal, 1980).”
Machine Comprehension with Syntax, Frames, and Semantics (2015, methodology): “To minimize the loss, we use the miniFunc package in MATLAB with LBFGS (Nocedal, 1980; Liu and Nocedal, 1989).”
A reliable effective terascale linear learning system (2011, methodology): “This solution ¯ w is used to initialize L-BFGS (Nocedal, 1980) with the standard Jacobi preconditioner.”
Constructing Unrestricted Adversarial Examples with Generative Models (2018, background): “[1] discovered that it is possible to find a slightly different image x 2 R such that kx xk  ✏ but f(x) 6= f(x), by solving a surrogate optimization problem with L-BFGS [28].”
Real-time cryo–EM data pre-processing with Warp (2018, methodology): “The cost and derivatives are used by the Limited-memory Broyden-Fletcher-Gold-farb-Shanno (L-BFGS) algorithm 26 to optimize the values of all control points.”
Shampoo: Preconditioned Stochastic Tensor Optimization (2018, methodology): “Classic algorithms in this family include Newton’s method, which employs the local Hessian as a preconditioner, as well as a plethora of quasi-Newton methods (e.g., [8, 15, 19]) that can be used whenever second-order information is unavailable or too expensive to compute.”
Automatic model construction with Gaussian processes (2014, methodology): “For all GP models, we fit kernel parameters by the standard method of maximizing training-set marginal likelihood, using L-BFGS (Nocedal, 1980) for 500 iterations, allowing five random restarts.”