The''echo state''approach to analysing and training recurrent neural networks

By Herbert Jaeger. 2001

3,264 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Deep learning in neural networks: An overview (2014, background): “Certain SL RNNs with fixed weights for all connections except those to output units (Jaeger, 2001; Maass et al., 2002; Jaeger, 2004; Schrauwen et al., 2007) have a maximal problem depth of 1, because only the final links in the corresponding CAPs are modifiable.”
Graph Neural Networks: A Review of Methods and Applications (2018, background): “Graph echo state network (GraphESN) (Gallicchio and Micheli, 2010) generalizes the echo state network (ESN) (Jaeger, 2001) on graphs.”
Harnessing Nonlinearity: Predicting Chaotic Systems and Saving Energy in Wireless Communication (2004, background): “ESNs can be applied to all basic tasks of signal processing and control, including time series prediction, inverse modeling, pattern generation, event detection and classification, modeling distributions of stochastic processes, filtering, and nonlinear control ( 16 , 18 , 19 , 20 ).”
An Empirical Exploration of Recurrent Network Architectures (2015, background): “…& Sutskever, 2011), regularization of the RNN's weights that ensures that the gradient does not vanish (Pascanu et al., 2012), giving up on learning the recurrent weights altogether (Jaeger & Haas, 2004; Jaeger, 2001), and a very careful initialization of RNN's parameters (Sutskever et al., 2013).”
Pretrained Transformers as Universal Computation Engines (2021, methodology): “2019) and echo state networks (Jaeger 2001; Jaeger and Haas 2004), a random recurrent network is frozen and only the output readout layer is trained.”
Metaheuristic design of feedforward neural networks: A review of two decades of research (2017, methodology): “A collection of RNN models, such as temporal RNN [29], echo state RNN [30], liquid state machine [31] and backpropagation de-correlation [32] forms a paradigm called reservoir computing, which addresses several engineering applications including nonlinear signal processing and control.”
No Training Required: Exploring Random Encoders for Sentence Classification (2019, methodology): “The echo state property (Jaeger, 2001) claims that the state of the reservoir should be determined uniquely from the input history, and the effects of a given state asymptotically diminish in favor of more recent states.”
Reservoir computing approaches to recurrent neural network training (2009, methodology): “, the largest absolute eigenvalue) satisfies ρ(W) < 1 [12].”
Using Conceptors to Manage Neural Long-Term Memories for Temporal Patterns (2017, background): “The task is to store two dissimilar periodic temporal patterns in a tiny RNN and subsequently retrieve them.”
Synaptic Plasticity Dynamics for Deep Continuous Local Learning (DECOLLE) (2018, background): “DECOLLE has some resemblance with reservoir networks, which are neural networks with fixed internal connectivity and trainable readout functions (Jaeger, 2001; Maass et al., 2002; Eliasmith and Anderson, 2004; Sussillo and Abbott, 2009).”
Recent Advances in Physical Reservoir Computing: A Review (2018, methodology): “It is also referred to as the echo state property, indicating that the influence of past inputs on the current reservoir states and outputs asymptotically fades out (Jaeger, 2001; Yildiz et al., 2012).”
Deep neural networks: a new framework for modelling biological vision and brain information processing (2015, background): “In an echo-state network (Jaeger 2001, see also Maass et al. 2002, for a similar model with spiking dynamics), for example, the sequence of input patterns is fed into a set of hidden units that are sparsely and randomly connected.”
Chaotic Time Series Prediction Based on a Novel Robust Echo State Network (2012, background): “The suggestion for the preparation of the reservoir can be referred in [18] and [29].”
Malware classification with recurrent networks (2015, background): “As a way of avoiding these issues, [14] introduced Echo State Networks, a form of recurrent neural models whose input and recurrent weights are not trained, but carefully sampled.”
Analyzing analytical methods: The case of phonology in neural models of spoken language (2020, background): “Two betterknown families of such techniques are called Echo State Networks (ESN) (Jaeger, 2001) and Liquid”
An overview of reservoir computing: theory, applications and implementations (2007, methodology): “Jaeger proposed that for ESNs the reservoir should be scaled such that they operate on the edge of stability, by setting the spectral radius 1 of the connection matrix close to, but less than one.”