Improving the convergence of back-propagation learning with second-order methods

By Suzanna Becker, Yann LeCun. 1989

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

ADADELTA: An Adaptive Learning Rate Method (2012, methodology): “Since we require the square root of this quantity in the parameter updates, this effectively becomes the RMS of previous squared gradients up to time t : ) where a constant (cid:15) is added to better condition the denominator as in [5].”
Fast and Furious Convergence: Stochastic Second Order Methods under Interpolation (2019, methodology): “In order to further scale these methods to high-dimensional datasets or to models with a large number of parameters, there has been a rich body of research to develop computationally efficient approximations to the Hessian matrix [4, 36, 46, 21, 56].”
Optimizing Neural Networks with Kronecker-factored Approximate Curvature (2015, methodology): “ the identity to the curvature matrix. Indeed, a subset of these techniques was exploited in the work of Martens (2010), and primitive versions of them have appeared implicitly in older works such as Becker and LeCun (1989), and also in many recent diagonal methods like that of Zeiler (2013), althou”
Deep learning in neural networks: An overview (2014, methodology): “The local gradient can be normalized based on the NN architecture (Schraudolph& Sejnowski, 1996), through a diagonalizedHessian approach (Becker & Le Cun, 1989), or related efficientmethods (Schraudolph, 2002).”
New Insights and Perspectives on the Natural Gradient Method (2014, methodology): “10 are often thought of as fudge factors designed to correct for the “poor conditioning” (Becker and LeCun, 1989) of the curvature matrix, or to guarantee boundedness of the updates and prevent the optimizer from “blowing up” (LeCun et al., 1998).”
Regularized estimation of image statistics by Score Matching (2010, methodology): “Here, we limit backpropagation to the diagonal approximation which scales like O(W ) [1].”
Using Noise to Compute Error Surfaces in Connectionist Networks: A Novel Means of Reducing Catastrophic Forgetting (2002, methodology): “Thus, unlike other techniques that make use of pseudopattern s which require the system to lear n a mixture of pseudopattern s and new patterns (Robins, 1995; French, 1997; Ans & Rousset, 1997, 2000 ), here noise is simply sent through the system, and this”
A Trace-restricted Kronecker-Factored Approximation to Natural Gradient (2020, unknown): “Many methods can be viewed as diagonal approximation of the curvature matrix (Becker and Lecun 1988; John, Elad, and Yoram 2011; Tieleman and Hinton 2012; Kingma and Ba 2014; Yao et al. 2020), and these algorithms are computationally tractable.”
Online Learning and Stochastic Approximations (1998, background): “Less eecient approximations have been designed (Becker and LeCun, 1989) and have proven eeective enough for large size applications (LeCun et al., 1989).”
Diagonal Rescaling For Neural Networks (2017, background): “This observation is important because most second-order optimization algorithms assume that the curvature changes slowly [Becker and LeCun, 1989, Martens and Grosse, 2015, Nocedal and Wright, 2006].”