Reservoir computing approaches to recurrent neural network training

By Mantas Lukosevicius, Herbert Jaeger. 2009

2,939 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Model Architectures, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

In Model Architectures

In Optimization & Training

Citations in the corpus

On the difficulty of training recurrent neural networks (2012, background): “As done in Martens and Sutskever (2011), we address the pathological problems proposed by Hochreiter and Schmidhuber (1997) that require learning long term correlations.”
Language Modeling Teaches You More Syntax than Translation Does: Lessons Learned Through Auxiliary Task Analysis (2018, methodology): “Our method of training auxiliary classiﬁers on randomly initialized RNNs builds on the tradition of reservoir computing, in which randomly initialized networks or “reservoirs” are ﬁxed and only “read-out” classiﬁer networks are trained (Luko ˇ sevi ˇ cius & Jaeger, 2009).”
Artificial Neural Networks-Based Machine Learning for Wireless Networks: A Tutorial (2017, background): “Note that, in theory, RNNs can make use of historical information in arbitrarily long sequences, but in practice they are limited to only a subset of the historical information [68].”
Re-visiting the echo state property (2012, background): “[8, 18]) it is crucial that the current network state xk is uniquely determined by any left-infinite input sequence .”
Information Processing Capacity of Dynamical Systems (2012, background): “ESNs are very successful at processing time dependent information 3,21 .”
A neurodynamical model for working memory (2011, background): “…2004; Lukosevicius, Jaeger, Popovici, & Siewert, 2007; Verstraeten, Schrauwen, & Stroobandt, 2006), is becoming a standard tool in recurrent neural network modeling (Jaeger, Maass, & Principe, 2007), and has spurred an active area of research (for a survey, see Lukosevicius and Jaeger (2009)).”
Sequence learning with hidden units in spiking neural networks (2011, methodology): “In contrast to Reservoir Computing [13] we train the weights towards hidden neurons which clearly helps to improve performance.”
Matching Recall and Storage in Sequence Learning with Spiking Neural Networks (2013, background): “Under the light of the remarkable capabilities of reservoir computing (Maass and Sontag, 2000; Lukos̆evic̆ius and Jaeger, 2009; Rigotti et al., 2010) it is interesting to note that a random choice of static hidden weights obtained by reshuffling learned weights is not sufficient to learn the target ”
ECG-based heartbeat classification for arrhythmia detection: A survey (2016, background): “More details regarding RC can be found in [124].”
Spatio-Temporal Graph Neural Networks for Predictive Learning in Urban Computing: A Survey (2023, background): “However, the biggest limitation of the original version of RNN is the gradient disappearance or gradient explosion problem that is difficult to solve during the training process [155].”
A Practical Guide to Applying Echo State Networks (2012, background): “RC today subsumes a number of related methods and extensions of the original idea [17], but the original ESN approach still holds its ground for its simplicity and power.”
Introduction to spiking neural networks: Information processing, learning and applications. (2011, background): “…Reservoir computing – is a computational concept that takes advantage of properties of recurrent networks, while avoiding difficulties associated with their training (Atiya and Parlos 2000, Dominey and Ramus 2000, Jaeger 2001, Maas et al. 2002a, Schrauwen et al. 2007, Lukosevicius and Jaeger 2009).”