Learning to Forget: Continual Prediction with LSTM

By Felix Alexander Gers, J. Urgen Schmidhuber, Fred Cummins. 2000

7,541 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

In Model Architectures

Citations in the corpus

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (2017, methodology): “Previous State-of-the-Art: The best previously published results (Jozefowicz et al., 2016) use models consisting of one or more stacked Long Short-Term Memory (LSTM) layers (Hochreiter & Schmidhuber, 1997; Gers et al., 2000).”
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation (2016, neutral): “Using a human side-by-side evaluation on a set of isolated simple sentences, it reduces translation errors by an average of 60% compared to Google’s phrase-based production system.”
Exploring the Limits of Language Modeling (2016, methodology): “To this extent, we used the Long-Short Term Memory model (Hochreiter & Schmidhuber, 1997) which uses a gating mechanism (Gers et al., 2000) to ensure proper propagation of information through many time steps.”
Semi-supervised Sequence Learning (2015, background): “Our LSTM implementation is standard and has input, for get, and output gates [6, 7].”
Training Very Deep Networks (2015, methodology): “To overcome this, we take inspiration from Long Short Term Memory (LSTM) recurrent networks [27, 28].”
Highway Networks (2015, methodology): “This scheme is strongly inspired by the proposal of Gers et al. (1999) to initially bias the gates in a Long Short-Term Memory recurrent network to help bridge long-term temporal dependencies early in learning.”
Recurrent World Models Facilitate Policy Evolution (2018, background): “While modern storage devices can store large amounts of historical data generated using an iterative training procedure, our LSTM [20, 34]- based world model may not be able to store all of the recorded information inside of its weight connections.”
Reinforcement Learning with Unsupervised Auxiliary Tasks (2016, methodology): “All agents used an LSTM with forget gates (Gers et al., 2000) with 256 cells which take in the CNN-encoded observation concatenated with the previous action taken and curren:t reward.”
The Best of Both Worlds: Combining Recent Advances in Neural Machine Translation (2018, methodology): “In addition to feeding the attention context to all decoder LSTM layers, we also feed it to the softmax.”
Generating Images from Captions with Attention (2015, methodology): “In a Bidirectional RNN, the two LSTMs (Hochre-iter & Schmidhuber, 1997) with forget gates (Gers et al., 2000)”
LSTM: A Search Space Odyssey (2015, background): “The vanilla LSTM incorporates changes by Gers et al. (1999) and Gers & Schmidhuber (2000) into the original LSTM (Hochreiter & Schmidhuber, 1997) and uses full gradient training.”
Transfer Learning for Low-Resource Neural Machine Translation (2016, methodology): “(2015) and Gers et al. (2000). For decoding we use a beam search of width 12.”
Gated Feedback Recurrent Neural Networks (2015, background): “Since the initial 1997 proposal, several variants of the LSTM have been introduced (Gers et al., 2000; Zaremba et al., 2014). Here we follow the implementation provided by Zaremba et al. (2014).”
Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations (2016, methodology): “In the most widely used formulation of an LSTM (Gers et al., 2000), ct and ht are computed via a set of four “gates”, including the forget gate, ft, which directly connects ct to the memories of the previous timestep ct−1, via an element-wise multiplication.”
Deep learning in neural networks: An overview (2014, methodology): “…Extraction of information conveyed by the temporal distance between events; Recognition of temporally extended patterns in noisy input sequences (Gers et al., 2000; Hochreiter & Schmidhuber, 1997b); Stable generation of precisely timed rhythms, as well as smooth and non-smooth periodic…”
Recent advances in convolutional neural networks (2015, methodology): “Inspired by Long Short Term Memory (LSTM) recurrent neural networks [72] which use gate functions to determine how much of a neuron’s activation value to transform or just pass through.”