Long Short-Term Memory in Echo State Networks: Details of a Simulation Study

By Herbert Jaeger. 2012

73 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

Citations in the corpus

On the difficulty of training recurrent neural networks (2012, background): “Jaeger (2012) also addresses this task with 100% success rate, though the solution does not seem to generalize well as it relies on very large output weights, which for ESNs are usually a sign of instability.”
Norm-preserving Orthogonal Permutation Linear Unit Activation Functions (OPLU) (2016, methodology): “Usually, architectural methods are used to prevent vanishing/exploding gradient in RNNs: NARX networks have short links between output error and old weights unfolded back in time [7], LSTM has a special structure of input and forgetting gates which produce a constant error carousel [8], for Echo Sta”
Learning-Based Methods for Comparing Sequences, with Applications to Audio-to-MIDI Alignment and Matching (2016, background): “, 2013; Graves, 2013), and/or careful initialization (Sutskever et al., 2013; Jaeger, 2012; Mikolov et al., 2014; Le et al., 2015) can help mitigate this issue and has facilitated the success of RNNs in a variety of fields (see e.”
Using Conceptors to Manage Neural Long-Term Memories for Temporal Patterns (2017, background): “• In computational neuroscience, attractor-like phenomena of all sorts are widely investigated as neural mechanisms for representing and processing information in neural systems (partial overviews in Durstewitz et al. (2000); Fusi and Wang (2016); Jaeger (2012)).”
A Practical Guide to Applying Echo State Networks (2012, background): “In some cases setting a small α, and thus inducing slow dynamics of x(n), can dramatically increase the duration of the short-term memory in ESN [22].”
Feed-Forward Networks with Attention Can Solve Some Long-Term Memory Problems (2015, methodology): “Slight variants of these tasks have also been used [7, 11, 9, 6], but we will follow the original definition from [3].”