On the limited memory BFGS method for large scale optimization

By Dong C. Liu, Jorge Nocedal. 1989

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

Fast large-scale optimization by unifying stochastic gradient and quasi-Newton methods (2013, methodology): “In BFGS and LBFGS, bad update detection is also performed, but it is achieved via a line search on the full objective.”
Do Better ImageNet Models Transfer Better? (2018, methodology): “We first examined the performance of different networks when used as fixed feature extractors by training an L2regularized logistic regression classifier on penultimate layer activations using L-BFGS [44] without data augmentation.”
Revisiting Self-Supervised Visual Representation Learning (2019, methodology): “Specifically, we precompute the visual representation for all training images and train the logistic regression using L-BFGS [27].”
Fast and Furious Convergence: Stochastic Second Order Methods under Interpolation (2019, methodology): “We prove that these algorithms, including the popular L-BFGS (Liu and Nocedal, 1989) (the limited-memory version of BFGS), can attain global linear convergence with a constant batch-size.”
Towards the Science of Security and Privacy in Machine Learning (2016, neutral): “The ﬁrst alters the training data either by inserting adversarial inputs into the existing training data (injection) or altering the training data directly (modiﬁcation).”
Optimization Methods for Large-Scale Machine Learning (2016, methodology): “trices, even when the exact Hessians are sparse, restricting its use to small and midsize problems. A common solution for this is to employ a limited memory scheme, leading to a method such as L-BFGS [97,113]. A key feature of this approach is that the matrices fH kgneed not be formed explicitly; in”
Neural Optimizer Search with Reinforcement Learning (2017, methodology): “More recent optimization methods combine insights from both stochastic and batch methods in that they use a small minibatch, similar to SGD, but implement many heuristics to estimate diagonal second-order information, similar to Hessian-free or L-BFGS (Liu & Nocedal, 1989).”
Spatially Transformed Adversarial Examples (2018, methodology): “We leverage L-BFGS (Liu & Nocedal, 1989) 0 1 2 3 4 5 6 7 8 stAdv on CIFAR-10 For CIFAR-10, we use ResNet-32 1 and wide ResNet-34 2 as the target clas-sifers (Zagoruyko & Komodakis, 2016; He et al., 2016; Madry et al., 2017).”
Federated Optimization: Distributed Machine Learning for On-Device Intelligence (2016, methodology): “These algorithms in general try to mimic the limited memory BFGS method (L-BFGS) [54], but model the local curvature information using inexact gradients — coming from the SGD procedure.”
Physics-Informed Neural Networks for Cardiac Activation Mapping (2020, methodology): “We train the network for 50,000 ADAM iterations with a batch size Nmb = 100 and then train with the L-BFGS method [28].”
Non-Linear Regression for Bag-of-Words Data via Gaussian Process Latent Variable Set Model (2015, methodology): “(11), we use the quasi-Newton method, which is a gradient-based optimization method (Liu and Nocedal 1989).”
Towards automated crystallographic structure refinement with phenix.refine (2012, methodology): “Coordinate refinement is performed using L-BFGS minimization (Liu & Nocedal, 1989) of the target Txyz (2) with respect to atomic positional parameters (individual coordinates or rotation– translation parameters of rigid bodies or torsion-angle space variables), while keeping all other parameters…”
Learning generative models for protein fold families (2011, methodology): “Parameter learning of the graph with just L 2 regularization can then be solved extremely efﬁciently using quasi-Newton methods [23].”