A Scaled Conjugate Gradient Algorithm for Fast Supervised Learning

By Martin Fodslette Møller. 1990

A supervised learning algorithm (Scaled Conjugate Gradient, SCG) with superlinear convergence rate is introduced. The algorithm is based upon a class of optimization techniques well known in numerical analysis as the Conjugate Gradient Methods. SCG uses second order information from the neural network but requires only O(N) memory usage, where N is the number of weights in the network. The performance of SCG is benchmarked against the performance of the standard backpropagation algorithm (BP), the conjugate gradient backpropagation (CGB) and the one-step Broyden-Fletcher-Goldfarb-Shanno memoryless quasi-Newton algorithm (BFGS). SCG yields a speed-up of at least an order of magnitude relative to BP. The speed-up depends on the convergence criterion, i.e., the bigger demand for reduction in error the bigger the speed-up. SCG is fully automated including no user dependent parameters and avoids a time consuming line-search, which CGB and BFGS use in each iteration in order to determine an appropriate step size. Incorporating problem dependent structural information in the architecture of a neural network often lowers the overall complexity. The smaller the complexity of the neural network relative to the problem domain, the bigger the possibility that the weight space contains long ravines characterized by sharp curvature. While BP is inefficient on these ravine phenomena, it is shown that SCG handles them effectively.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

A data-driven approach to predict the success of bank telemarketing (2014, methodology): “For DT, rminer adopts the default parameters of the rpart R package,which implements the popular CART algorithm [2] For the LR and NN learning, rminer uses the efficient BFGS algorithm [22], from the family of quasi-Newtonmethods, while SVM is trained using the sequential minimal optimization (SMO) ”
A hybrid model for high-frequency stock market forecasting (2015, methodology): “…adjustment procedure (PAP), for ANN models (multilayer perceptron like) built using a hybrid learning process composed by GA (Leung et al., 2003) and gradient-based techniques (Battiti, 1992; Hagan & Menhaj, 1994; Moller, 1993; Riedmiller & Braun, 1993), which solves of a practical way the RWD.”
Evolving artificial neural networks (1999, methodology): “Evolutionary training can be slow for some problems in comparison with fast variants of BP [131] and conjugate gradient algorithms [19], [132].”
A hybrid neural network and ARIMA model for water quality time series prediction (2010, methodology): “On of the optimized methods developed by Moller (1993) is the scaled conjugate gradient (SCG) algorithm.”
Bayesian Optimization and Semiparametric Models with Applications to Assistive Technology (2014, methodology): “This is why Lawrence [2005] advocates the use of scaled conjugate gradients [Møller, 1993].”
Group Search Optimizer: An Optimization Algorithm Inspired by Animal Searching Behavior (2009, methodology): “For comparison, we also employed an improved gradientbased training algorithm: scaled conjugate gradient (SCG) backpropagation [52]; and four EA-based training algorithms, namely, simple genetic algorithm (SGA) [35] based algorithm; evolutionary programming (EP) [36], [37] based algorithm; evolution”
A Stochastic Quasi-Newton Method for Online Convex Optimization (2007, methodology): “Online implementations of conjugate gradient methods (Møller, 1993; Schraudolph and Graepel, 2003) have therefore proven largely ineffective.”
Predictive Abilities of Bayesian Regularization and Levenberg–Marquardt Algorithms in Artificial Neural Networks: A Comparative Empirical Study on Social Data (2016, background): “Essentially, the structure of an ANN is computer-based and consists of several simple processing elements operating in parallel [3,5,6].”