Representations of quasi-Newton matrices and their use in limited memory methods

By Richard H. Byrd, Jorge Nocedal, Bobby Schnabel. 1994

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

DeltaGrad: Rapid retraining of machine learning models (2020, methodology): “Then it uses an effcient L-BGFS update from Byrd et al. (1994) (see Appendix A.2.1 for the details of the L-BGFS update).”
The State of the Art (2001, unknown): “Hk = A −1 k there exists for the case A0 = 1 γ I another closed formula [24]”
An Introduction to Conditional Random Fields for Relational Learning (2007, methodology): “A full K × K approximation to the Hessian still requires quadratic size, however, so a limited-memory version of BFGS is used, due to Byrd et al. [1994].”
Convergent Decomposition Solvers for Tree-reweighted Free Energies (2011, background): “…model subject to the original constraints: min θ∈C(θ) ˜ Φ (k) +( θ− θ (k) )·∇ (k) θ + 1 2 ( θ− θ (k) ) T B (k) ( θ− θ (k) ) where B (k) is a positive-definitive approximation to the Hessian that is maintained in compact form in terms of the previous p iterates and gradients (Byrd et al., 1994).”
Conditional Neural Fields (2009, methodology): “In this work, we use the logistic function as the gate function.”
Projected Newton-type methods in machine learning (2011, methodology): “By taking the compact representation of Byrd et al. (1994)”
Table extraction using conditional random fields (2003, methodology): “As with the CRF, we use a Gaussian prior, and train using L-BFGS.”
Coupled Group Lasso for Web-Scale CTR Prediction in Display Advertising (2014, background): “Hence, it’s more suitable for optimization problems with a large number of variables (Nocedal, 1980; Byrd et al., 1994).”
A Distributed Quasi-Newton Algorithm for Empirical Risk Minimization with Nonsmooth Regularization (2018, background): “et al. [4], given a prespecified integer m > 0, at the t th iteration for t > 0, let ˜ m (cid:66) min ( m , t ) , and define At the first iteration where no s i and y i are available, we set H a I for some positive scalar a 0 When f is twice-differentiable and If f is not strongly convex, it is…”
Composition of Conditional Random Fields for Transfer Learning (2005, unknown): “Particularly effective are gradientbased methods that use approximate second-order information, such as conjugate gradient and limited-memory BFGS (Byrd et al., 1994).”
Efficiently Inducing Features of Conditional Random Fields (2002, methodology): “, 1997), however it has recently been shown that quasi-Newton methods, such as L-BFGS, are significantly more efficient (Byrd et al., 1994; Malouf, 2002; Sha & Pereira, 2003).”
Quasi-Newton Methods on Grassmannians and Multilinear Approximations of Tensors (2009, methodology): “We give a brief summary of the limited memory quasi-Newton method with L-BFGS updates on Euclidean spaces [6] that we need later for our Grassmann variant.”
Numerical optimization (2004, methodology): “24), that provides a compact representation of the inverse BFGS approximation Hk ; see [52] for details.”
An interior algorithm for nonlinear optimization that combines line search and trust region steps (2006, methodology): “We have implemented a limited memory BFGS method using the compact representations described in [8].”
Quasi-Newton Methods for Deep Learning: Forget the Past, Just Sample (2019, methodology): “• Implementing the limited memory SR1 methods is not trivial; we made use of the compact representations of the SR1 matrices [12] and computed the steps dynamically; see Appendix B.”
Optimization Methods for Supervised Machine Learning: From Linear Models to Deep Learning (2017, background): “Fortunately, however, the optimization literature has already addressed these latter issues with the idea of using limited memory quasi-Newton approaches, such as limited memory BFGS, also known as L-BFGS [13].”