On Restart Procedures for the Conjugate Gradient Method

By Yuhong Dai, L. Liao, Duan Li. 2004

622 citations recorded by the source corpus.

Subfields: Computational Theory and Mathematics, Numerical Analysis, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Theory and Mathematics

In Numerical Analysis

In Optimization & Training

Citations in the corpus

SGDR: Stochastic Gradient Descent with Warm Restarts (2016, methodology): “This scheme resembles the one of Powell (1977) for the conjugate gradient method.”
A New Conjugate Gradient Method with Guaranteed Descent and an Efficient Line Search (2005, neutral): “The Perry–Shanno scheme, analyzed further in [34, 37, 39], has global convergence for convex functions and for an inexact line search [36], but in general, it does not necessarily converge, even when the line search is exact [33].”
A Nonlinear Conjugate Gradient Algorithm with an Optimal Property and an Improved Wolfe Line Search (2013, neutral): “Since for any τ k , β k ( τ k ) = β P RP k if g T k+1 d k = 0 , Powell ' s example can also be used to show the method ( 1 .”
Approximate Riemannian Conjugate Gradient Learning for Fixed-Form Variational Bayes (2010, methodology): “We also apply a Riemannian version of the Powell-Beale restart method (Powell, 1977): the search direction is reset to the negative gradient direction if |˜g k−1 , ˜ g k k | ≥ 0.2||˜g k || 2 k .”
A modified Fletcher-Reeves-Type derivative-free method for symmetric nonlinearequations (2011, background): “Other good results about the convergence property of the FR method and its improvement can be found in [5, 7, 9, 11, 13, 16, 21, 22] etc.”
Hardware implementation of memristor-based artificial neural networks (2024, neutral): “…methods which avoids the use of the Hessian matrix and instead makes use of the conjugated direction of the gradient, e.g. Scaled Conjugate Gradient 173 , Conjugate Gradient with Powell-Beale restarts 174 , Fletcher-Powell Conjugate Gradients 175 and Polak-Ribiere Conjugate Gradient 165,175 ).”
A SURVEY OF NONLINEAR CONJUGATE GRADIENT METHODS (2005, methodology): “In order to ensure that the search directions generated by this family will be descent directions, Powell’s [88] restart criterion is employed: set dk = −gk if”
Loss of plasticity in deep continual learning (2024, neutral): “The variation and selection is reminiscent of trial-and-error processes in evolution and behaviour 44–47 and has precursors in many earlier ideas, including Keifer–Wolfowitz methods 48 and restart methods 49 in engineering and feature-search methods in machine learning 31–35 , 50 .”