Automatic Learning Rate Maximization by On-Line Estimation of the Hessian's Eigenvectors

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Partial BFGS Update and Efficient Step-Length Calculation for Three-Layer Neural Networks (1997, methodology): “To overcome these drawbacks, a learning rate maximization method (LeCun et al. 1993), learning rate adaptation rules (Jacobs 1988; Silva and Almeida 1990), smart algorithms such as QuickProp (Fahlman 1988) and RPROP (Riedmiller and Braun 1993), and second-order learning algorithms (Watrous 1987; Bar”
Learning Rate Adaptation in Stochastic Gradient Descent (2001, methodology): “According to experiments reported in [11], the largest eigenvalue of the Hessian is mainly determined by the BPNN’s architecture, the initial weights and by short–term low–order statistics of the training data.”
Fast Exact Multiplication by the Hessian (1994, methodology): “This approximation was used to good effect in Le Cun et al. (1993) and in many numerical analysis optimization routines, which use it to gradually build up an approximation to the inverse Hessian.”
Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability (2021, background): “This conventional wisdom (LeCun et al., 1993; 1998; Schaul et al., 2013) holds that if the sharpness at step t is λt, then the current step size ηt must be set no greater than 2/λt (in order to prevent divergence); and furthermore, barring additional information about the objective function, that…”