First- and Second-Order Methods for Learning: Between Steepest Descent and Newton's Method

By Roberto Battiti. 1992

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Deep learning in neural networks: An overview (2014, methodology): “Least-squares methods (Gauss– Newton, Levenberg–Marquardt) (Gauss, 1809; Levenberg, 1944; Marquardt, 1963; Newton, 1687; Schaback & Werner, 1992) and quasi-Newtonmethods (Broyden–Fletcher–Goldfarb–Shanno, BFGS) (Broyden et al., 1965; Fletcher & Powell, 1963; Goldfarb, 1970; Shanno, 1970) are computa”
Neural networks for classification: a survey (2000, background): “Other research topics related to neural classification include network training [12], [15], [62], [124], [142], model design and selection [50], [72], [117], [121], [122], [180], [194], sample size issues [51], [135], [136], Bayesian analysis [102], [109], [110], [120], and wavelet networks [165],…”
Recurrent neural networks (2013, background): “In particular, it generally does not suffer from the problem of slow convergence (Battiti, 1992; Hagan & Menhaj, 1994) and can yield a good cost function compared with the other training algorithms.”
Hardware implementation of memristor-based artificial neural networks (2024, unknown): “Alternatively, other methods are the Levenberg-Marquardt 176 (uses the Jacobian matrix instead of the Hessian Matrix), Resilient Backpropagation 177 and One Step Secant 178 , but these are more demanding from a computational point of view.”
Special-purpose digital hardware for neural networks: An architectural survey (1996, methodology): “For instance, to improve over simple back-propagation (BP) for multilayer perceptrons (MLP) [ 16], conjugate gradient and second order methods have been suggested [ 17-19 ].”
Recurrent neural networks employing Lyapunov exponents for EEG signals classification (2005, background): “(4), Jk is the Jacobian of f evaluated at wk, l is the Marquardt parameter, I is the identity matrix (Battiti, 1992; Hagan & Menhaj, 1994).”
Revisiting Recursive Least Squares for Training Deep Neural Networks (2021, methodology): “The best known second-order algorithm should be the Newton’s algorithm [15].”
A comprehensive review for industrial applicability of artificial neural networks (2003, methodology): “Although the back-propagation algorithm has been a significant milestone, many attempts have been made to speed up the convergence and significant improvement are observed by using various second order approaches, namely, Newton’s method, conjugate gradient’s, or the LM optimization technique [5], [”