The Conjugate Gradient Method and Trust Regions in Large Scale Optimization

By Trond Steihaug. 1983

Subfields: Mathematical Optimization, Numerical Methods & Scientific Computing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Mathematical Optimization

In Numerical Methods & Scientific Computing

Citations in the corpus

Optimization Methods for Large-Scale Machine Learning (2016, background): “ a line search and to introduce an additional condition in Step5of Algorithm6.1: terminate CG if a candidate solution s k is a direction of negative curvature, i.e., sT k r 2f SH k (w k;˘H k )s <0 [142]. A number of more sophisticated strategies have been proposed throughout the years with some s”
New Insights and Perspectives on the Natural Gradient Method (2014, methodology): “The exact minimizer 10 δ∗ = −G(θk)−1∇h(θk) is often too difficult to compute, and so practical methods will often only approximately minimize Mk(δ) (e.g Dembo et al., 1982; Steihaug, 1983; Dennis Jr and Schnabel, 1996; Martens, 2010; Vinyals and Povey, 2012).”
Recursive Trust-Region Methods for Multiscale Nonlinear Optimization (2008, methodology): “In this section, we only consider V-cycle iterations, where minimization at lower levels (above the coarsest) consists in at most one successful smoothing iteration followed by either a successful TCG Taylor iteration (if (2.14) fails) or a recursive iteration (if (2.14) holds), itself followed by a”
On the Use of Stochastic Hessian Information in Optimization Methods for Machine Learning (2011, background): “For k ¼ 0; 1; : : : , until a convergence test is satisfied: 1.”
A Newton-CG Augmented Lagrangian Method for Semidefinite Programming (2010, methodology): “As pointed out by one of the referees, for globalizing the semismooth NewtonCG algorithm, one may also use Steihaug’s trust region truncated CG approach [34] instead of the line search truncated CG approach.”
Trust-Region Methods on Riemannian Manifolds (2007, methodology): “The concept of truncated CG is due to Steihaug [ Ste83 ] and Toint [Toi81]; therefore the method is sometimes referred to as the Steihaug-Toint algorithm, see e.g.”
Adaptive norms for deep learning with regularized Newton methods (2019, methodology): “[60] Trond Steihaug.”
Mining adversarial patterns via regularized loss minimization (2010, background): “The problem in Equation 13 is also known as the trust region sub-problem [14].”
Iterative Scaled Trust-Region Learning in Krylov Subspaces via Pearlmutter's Implicit Sparse Hessian-Vector Multiply (2003, background): “…real-life large-scale problems to find out the strengths of the full-Hessian based methods (see Algorithm E) perhaps with a more elaborate preconditioner, which would be much more time-consuming per epoch but may reduce the total time dramatically; hence, need to deal with a delicate balancing act.”
On the limited memory BFGS method for large scale optimization (1989, methodology): “Among the most useful methods for solving this problems are: (i) Newton's method and variations of it; see for example Steihaug (1983), O'Leary (1982), Toint (1981) and Nash (1985); (ii) the partitioned quasi-Newton method of Griewank and Toint (1982a); (iii) the conjugate gradient method; see for…”
Solving the Trust-Region Subproblem using the Lanczos Method (1999, methodology): “Such a strategy was first proposed independently by Steihaug [22] and Toint [23], and we shall refer to the terminating point as the Steihaug–Toint point.”
A Generalized Approach to Portfolio Optimization: Improving Performance by Constraining Portfolio Norms (2009, methodology): “Then, it trivially follows from the analysis in Phatak and De Hoog (2003) and Steihaug (1983) that the 2-norm of the iterates generated by the conjugate-gradient method is nondecreasing.”
Structure of the Hessian matrix and an economical implementation of Newton’s method in the problem of canonical approximation of tensors (2010, methodology): “3) are as follows (see [13]).”