Recurrent nets that time and count

By Felix Alexander Gers, J. Urgen Schmidhuber. 2000

666 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Electrical and Electronic Engineering, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Electrical and Electronic Engineering

In Model Architectures

Citations in the corpus

Light Gated Recurrent Units for Speech Recognition (2018, background): “purpose RNNs such as Long Short Term Memories (LSTMs) [28] have been the subject of several studies and modifications over the past years [7], [29], [30].”
LSTM: A Search Space Odyssey (2015, background): “Gers and Schmidhuber [22] argued that in order to learn”
Federated Learning Of Out-Of-Vocabulary Words (2019, methodology): “In this work we use a variant of LSTM with a Coupled Input and Forget Gate (CIFG) (Greff et al., 2017), peephole connections (Gers and Schmidhuber, 2000) and a projection layer (Sak et al., 2014).”
Deep learning in neural networks: An overview (2014, background): “smooth and non-smooth periodic trajectories (Gers and Schmidhuber, 2000).”
RUDDER: Return Decomposition for Delayed Rewards (2018, neutral): “s vanishing gradients and allows for uniform credit assignment. The most commonly used LSTM memory cell architecture in the literature [33, 95] contains forget gates [25, 26] and peephole connections [24]. In our previous work [49, 45], we found that peephole connections are only useful for modeling”
Learning to Diagnose with LSTM Recurrent Neural Networks (2015, methodology): “Our proposed LSTM RNN uses memory cells with forget gates (Gers et al., 2000) but without peephole connections (Gers et al., 2003).”
Audio Word2Vec: Unsupervised Learning of Audio Segment Representations Using Sequence-to-Sequence Autoencoder (2016, methodology): “We adopted the LSTM described in [34], and the peephole connections [35] were added.”
Reinforcement Learning Upside Down: Don't Predict Rewards - Just Map Them to Actions (2019, background): “If there is only one single lifelong trial, however, this argument does not hold for the RNN-based C of Algorithms B1/B2, because at each time step, an RNN could in principle uniquely represent the entire history so far, for instance, by learning to simply count the time steps [10].”
Explaining and Interpreting LSTMs (2019, background): “forget gate f, and gets accumulated through a sum with the current input. 4 Arras et al. LSTM memory cell architecture in the literature [25,66] contains forget gates [19,20] and peephole connections [18]. In our previous work [38,34], we found that peephole connections are only useful for modeling ”
A Critical Review of Recurrent Neural Networks for Sequence Learning (2015, background): “Also in 2000, Gers and Schmidhuber proposed peephole connections [Gers and Schmidhuber, 2000], which pass from the internal state directly to the input and output gates of that same node without first having to pass through the output gate.”
Machine learning for estimation of building energy consumption and performance: a review (2018, methodology): “This type of neural network usually employed to solve very deep learning tasks (i.e. more than 1000 layers are needed) (Pérez-Ortiz et al. 2003; Gers and Schmidhuber 2000).”
DeepSleepNet: A Model for Automatic Sleep Stage Scoring Based on Raw Single-Channel EEG (2017, methodology): “We also use peephole connections [19], [20] in our LSTMs which allow their gating mechanism to inspect their current memory cell before the modification.”
The History Began from AlexNet: A Comprehensive Survey on Deep Learning Approaches (2018, methodology): “On the other hand, LSTM provides better performance, if you have enough data and computational power [137].”
Measuring the Information Content of Financial News (2016, positive): “We take a variation of LSTM with peephole connections (Gers and Schmidhuber, 2000), which uses a input gate it, a forget gate ft and a output gate ot in the same memory block to learn from the current cell state.”
Unsupervised Learning in LSTM Recurrent Neural Networks (2001, methodology): “This permits LSTM to bridge huge time lags (1000 steps and more) between relevant events, while traditional RNNs already fail to learn in the presence of 10-step time lags, despite requiring more complex update algorithms. See [ 52 ] for a detailed description.”
Music transcription modelling and composition using deep learning (2016, result): “This is consistent with previous ﬁndings about RNN [14].”