Optimal filtering algorithms for fast learning in feedforward neural networks

By S. Shah, F. Palmieri, M. Datum. 1992

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Long-term wind speed and power forecasting using local recurrent neural network models (2006, methodology): “The algorithm is identical to a nonlinear recursive least squares (RLS) [ 22 ] method that minimizes the error criterion”
Neural Networks: Tricks of the Trade (2012, methodology): “Several attempts using “mini batches” in applying conjugate gradient to large and redundant problems have been made recently [17, 25, 31].”
Sequential Monte Carlo Methods to Train Neural Network Models (2000, methodology): “Shah, Palmieri, and Datum (1992) and Puskorius and Feldkamp (1991; Puskorius, Feldkamp, & Davis, 1996) have proposed strategies for decoupling the global EKF estimation algorithm into local EKF estimation subproblems.”
Centering Neural Network Gradient Factors (1996, background): “This is known to sometimes improve convergence behavior [11], and we have found it to increase stability at the large step sizes we desire.”
Bayesian methods for neural networks (2000, background): “Shah, Palmieri and Datum (Shah et al., 1992) and Puskorius and Feldkamp (Puskorius and Feldkamp, 1991) have proposed various approximations to the weights covariance so as to simplify this problem.”
A unified theory of adaptive stochastic gradient descent as Bayesian filtering (2018, methodology): “Here we provide such a theory by returning to very early work that used Bayesian (Kalman) ﬁltering to optimize the parameters of neural networks (Puskorius & Feldkamp, 1991; Sha et al., 1992; Puskorius & Feldkamp, 1994; 2001; Feldkamp et al., 2003).”
Bayesian filtering unifies adaptive and non-adaptive neural network optimization methods (2018, methodology): “Here we provide such a theory by reconciling state-of-the-art adaptive SGD algorithms with very early work that used Bayesian (Kalman) filtering to optimize the parameters of neural networks (Puskorius & Feldkamp, 1991; Sha et al., 1992; Puskorius & Feldkamp, 1994, 2001; Feldkamp et al., 2003; Olliv”
Continuous Learning of Context-dependent Processing in Neural Networks (2018, methodology): “is equivalent to use RLS to train neural networks under the name of Enhanced Back Propagation (EBP), which was proposed to increase the speed of convergence in training [27].”
Local Gain Adaptation in Stochastic Gradient Descent (1999, methodology): “We evaluated our work on the “four regions” classification task [27], a well-known benchmark problem [17, 28, 29, 30]: a fully connected feedforward network with 2 hidden layers of 10 units each (tanh nonlinearity) is to classify two continuous inputs (range [-1,1]) as illustrated in Figure 1.”