LSTM can Solve Hard Long Time Lag Problems

By Sepp Hochreiter, J. Urgen Schmidhuber. 1996

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

Citations in the corpus

Sequence to Sequence Learning with Neural Networks (2014, background): “As a result, the problem has a large “minimal time lag” [17].”
Methods for interpreting and understanding deep neural networks (2017, background): “Examples are LSTMs [26], normalization layers [27,33], or improved element-wise nonlinearities [31].”
RUDDER: Return Decomposition for Delayed Rewards (2018, background): “ell output (A331) A8.3 Long-Term Dependencies vs. Uniform Credit Assignment The LSTM network has been proposed with the aim to learn long-term dependencies in sequences which span over long intervals [55, 56, 50, 51]. However, besides extracting long-term dependencies, LSTM memory cells have another”
Textual Explanations for Self-Driving Vehicles (2018, methodology): “For an image classiﬁcation problem, [7,8] used an LSTM [9] caption generation model that generates textual justiﬁcations for a CNN model.”
Why Neural Translations are the Right Length (2016, methodology): “In this paper, we train long shortterm memory (LSTM) neural units (Hochreiter and Schmidhuber, 1997) trained with back-propagation through time (Werbos, 1990).”
Explaining and Interpreting LSTMs (2019, methodology): “The LSTM network has been proposed with the aim to learn long-term dependencies in sequences which span over long intervals [37,36,31,32].”
Training Recurrent Networks by Evolino (2007, background): “As we showed [12, 39], many RNN pro blems involving long-term dependencies that were considered challengi ng benchmarks in the 1990s, turned out to be trivial in that they could be solved by random weight guessing.”
A Review of Deep Learning Methods and Applications for Unmanned Aerial Vehicles (2017, background): “Long Short-Term Memory (LSTM) models are a type of RNN architecture proposed in 1997 by Hochreiter and Schmidhuber [17] which successfully overcomes the problem of vanishing gradients by maintaining a more constant error through the use of gated cells, which effectively allow for continuous learning”
MolGenSurvey: A Systematic Survey in Machine Learning Models for Molecule Design (2022, methodology): “ARs have been widely-used in generation tasks, including recurrent neural network (RNN) [80], [81], PixelCNN [82], WaveNet [83], Transformer [37] and BERT [38].”
A review on the long short-term memory model (2020, background): “The LSTM model (Hochreiter and Schmidhuber 1997a) is a powerful recurrent neural system specially designed to overcome the exploding/vanishing gradient problems that typically arise when learning long-term dependencies, even when the minimal time lags are very long (Hochreiter and Schmidhuber…”
Artificial intelligence in recommender systems (2020, background): “Representative RNN is a long short-term memory (LSTM) network [37] which is suitable for time series prediction.”
On the Robustness of Self-Attentive Models (2019, methodology): “For classiﬁcation tasks including sentiment analysis and entailment detection, we use a Bidi-rectional LSTM with an attention (Hochreiter and Schmidhuber, 1997; Bahdanau et al., 2014) layer as the sentence encoder, and a fully connected layer for classiﬁcation problems.”
Sentiment Analysis Based on Deep Learning: A Comparative Study (2020, methodology): “s. A special type of RNN is long short-term memory (LSTM), which is capable of using long memory as the input of activation functionsinthehiddenlayer. ThiswasintroducedbyHochreiterandSchmidhuber(1997)[24]. Figure4 illustrates an example of the LSTM architecture. The input data is preprocessed to res”
CatBoost for big data: an interdisciplinary review (2020, methodology): “report, CatBoost outperforms Random Forest [73], Long Short-term Memory ( LSTM ) [83], and Seq2Seq [84].”
Adaptive RNN Tree for Large-Scale Human Action Recognition (2017, background): “This problem is alleviated with the long short-term memory (LSTM) cells [10] as a replacement of the traditional nonlinear units.”
Generative artificial intelligence: a systematic review and applications (2024, unknown): “ELMo utilizes a bidirectional LSTM (Long Short-Term Memory) network [47] for contextual word embeddings.”
When Are Tree Structures Necessary for Deep Learning of Representations? (2015, background): “For instance, Tai et al. (2015) found that deep LSTM tree models outperform deep LSTM sequence models on the Stanford SentimentBank task, suggesting that may offer different or greater advantages in recursive models.”
A language-independent neural network for event detection (2016, unknown): “Additionally, compared with the standard RNN, Bi-LSTM does not face the problem of gradient vanishing or exploding [30].”