New results on recurrent network training: unifying the algorithms and accelerating convergence

By Amir F. Atiya, Alexander G. Parlos. 2000

538 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computational Mathematics, Model Architectures, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mathematics

In Model Architectures

In Optimization & Training

Citations in the corpus

On the difficulty of training recurrent neural networks (2012, methodology): “ been proposed in the literature, like Backpropagation Through Time (BPTT) (Rumelhart et al., 1986; Werbos, 1988), Real Time Recurrent Learning (Williams and Zipser, 1989), AtiyaParlos learning rule (Atiya and Parlos, 2000), etc. Most of these are gradient based, though alternative approaches are al”
Deep learning in neural networks: An overview (2014, neutral): “See generalizations for sequenceprocessing recurrent NNs (e.g., Atiya & Parlos, 2000; Baldi, 1995; Gherrity, 1989; Kremer & Kolen, 2001; Pearlmutter, 1989, 1995; Robinson & Fallside, 1987; Rohwer, 1989; Schmidhuber, 1992a; Werbos, 1988;Williams, 1989;Williams & Peng, 1990;Williams & Zipser, 1988,…”
Reservoir computing approaches to recurrent neural network training (2009, background): “It approximates and signiﬁcantly simpliﬁes the APRL method, and applies it only to the output weights W out , turning it into an online RC method.”
Towards a theoretical foundation for morphological computation with compliant bodies (2011, methodology): “The systems 7 and 8 were both taken from Atiya and Parlos (2000), where they were used in order to demonstrate the performance of a new learning algorithm for recurrent networks.”
Reservoir Computing Trends (2012, methodology): “In an efficient version of gradient descent RNN training introduced by Atiya and Parlos [1] the output weights W are adapted much more than W and W [38], which led to a further optimization where they remain constant, an RC method called BackPropagation-DeCorrelation (BPDC) [43].”
Federated learning for 6G communications: Challenges, methods, and future directions (2020, methodology): “…Level : At the algorithm level, achieving communication-efﬁcient FL can reduce the communication rounds of training a model by accelerating convergence [25] and reduce the communication cost of each round by using gradient compression techniques [10] (e.g., sparsiﬁcation, quantization, etc.).”
Adaptive Nonlinear System Identification with Echo State Networks (2002, background): “!] error obtained in [1] on a length 200 training sequence was NMSEtrain ~ 0.2412 However, the level of precision reported [1] and many other published papers about RNN training appear to be based on suboptimal training schemes.”
An overview of reservoir computing: theory, applications and implementations (2007, background): “Yet other research by Steil [53] showed that the current state-of-the-art learning rule for RNNs (Athya-Parlos recurrent learning [1]) actually has the same weight dynamics as what was proposed by Jaeger and Maass: the outputs weights are trained, and the internal weights are only globally scaled…”
Optoelectronic Reservoir Computing (2011, methodology): “Two variants are widely used: a 10th order system (NARMA10) introduced in [28], and the more difficult 30th order system (NARMA30) (see for instance [35] for a definition).”
A Dynamical System-based Approach to Modeling Stable Robot Control Policies via Imitation Learning (2012, methodology): “in case of reaching motion), bifurcation during learning, and complexities of existing advanced training algorithms (Hopfield, 2007; Atiya et al., 2000; Lukosevicius & Jaeger, 2009).”
Introduction to spiking neural networks: Information processing, learning and applications. (2011, background): “Many implementations of unsupervised learning in perceptron-or sigmoidal-gate-based ANN have been applied to real-world tasks.”
Generating Coherent Patterns of Activity from Chaotic Neural Networks (2009, unknown): “…procedures have also been developed for recurrently connected neural networks (Rumelhart et al., 1986; Williams and Zipser, 1989; Pearlmutter, 1989; Atiya and Parlos, 2000), but these are more computationally demanding and difficult to use than feedforward learning algorithms, and there are…”
Learning control Lyapunov function to ensure stability of dynamical system-based robot reaching motions (2014, background): “An overview of different RNN structures and training algorithms are presented in [47].”
The''echo state''approach to analysing and training recurrent neural networks (2001, background): “Several basic algorithms are known and have been refined in various directions (overviews in [3], [15], [2]).”