Training Recurrent Networks by Evolino

By J. Urgen Schmidhuber, Daan Wierstra, Matteo Gagliolo, Faustino J. Gomez. 2007

286 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

Citations in the corpus

Weight Agnostic Neural Networks (2019, background): “Schmidhuber et al. [96] have shown that a randomly-initialized LSTM [45] with a learned linear output layer can predict time series where traditional RNNs fail.”
Deep learning in neural networks: An overview (2014, methodology): “Some are better solved by using linear methods to obtain optimal weights for connections to output events (Section 2), and evolving weights of connections to other events—this is called Evolino (Schmidhuber, Wierstra, Gagliolo, & Gomez, 2007).”
Evolution Strategies as a Scalable Alternative to Reinforcement Learning (2017, background): “, 2008; 2014), as well as for the training of the recurrent weights in recurrent neural networks (the hidden-to-output weights have been trained with supervised learning; Schmidhuber et al., 2007).”
Understanding Deep Architectures using a Recursive Convolutional Network (2013, background): “These are are well-studied models [11, 21, 27], naturally suited to temporal and sequential data.”
On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models (2015, positive): “Those early CM systems, however, did not yet use powerful RNNs such as LSTM.”
Flipout: Efficient Pseudo-Independent Weight Perturbations on Mini-Batches (2018, methodology): “ES was recently proposed as an alternative reinforcement learning algorithm (Schmidhuber et al., 2007; Salimans et al., 2017).”
Audio Word2Vec: Unsupervised Learning of Audio Segment Representations Using Sequence-to-Sequence Autoencoder (2016, positive): “Because many amazing results were achieved by LSTM, RNN has widely been equipped with LSTM units [25, 26, 27, 28, 22, 29, 30].”
Agent Inspired Trading Using Recurrent Reinforcement Learning and LSTM Neural Networks (2017, methodology): “…estimation of distribution algorithms, particle swarm optimization [26], the cross-entropy method [27], covariance matrix adaptation evolution strategy CMA-ES [28], which is considered by many to be the ”industry standard” of evolutionary computation, Natural Evolution Strategies, and Evolino[29].”
Minimum Complexity Echo State Network (2011, background): “Sometimes reservoirs have been evolve d in a costly and difficult to analyze evolutionary computatio n setting [8], [14], [41], [42].”
Advanced monitoring of machining operations (2010, background): “Further supervised NN approaches are also considered here due to their use in decision making during monitoring of machining: probabilistic NN (PNN) [185], recurrent NN (RNN) [186–188], artificial cellular NN (ACNN) [189], fuzzy logic NN (FLNN) or neurofuzzy systems (NFS) combiningNN and FLmethods t”
Reservoir computing approaches to recurrent neural network training (2009, background): “Evolino [46] transfers the idea of ESNs from an RNN of simple sigmoidal units to a Long Short-Term Memory type of RNNs [40] constructed from units capable of preserving memory for long periods of time.”
Evolutionary Computation for Reinforcement Learning (2012, methodology): “Hybrid methods have also been employed to improve performance on supervised learning tasks (Gruau and Whitley, 1993; Boers et al, 1995; Giraud-Carrier, 2000; Schmidhuber et al, 2005, 2007).”
A review on the long short-term memory model (2020, background): “Alternatively, one can train the model with evolino (Schmidhuber et al. 2007; Wierstra et al. 2005).”
Supervised Sequence Labelling with Recurrent Neural Networks (2012, background): “Recently, non gradient-based training methods of LSTM have also been considered (Wierstra et al., 2005; Schmidhuber et al., 2007), but they are outside the scope of this book.”