Long Short-Term Memory

By Sepp Hochreiter, J. Urgen Schmidhuber, J. Schmidhuber. 1997

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

Citations in the corpus

Attention is All you Need (2017, background): “Recurrent neural networks, long short-term memory [12] and gated recurrent [7] neural networks in particular, have been firmly established as state of the art approaches in sequence modeling and transduction problems such as language modeling and machine translation [29, 2, 5].”
Deep Residual Learning for Image Recognition (2015, background): “Concurrent with our work, “highway networks” [42, 43] present shortcut connections with gating functions [15].”
A Deep Reinforced Model for Abstractive Summarization (2017, positive): “We use a single LSTM decoder RNN d , computing hidden states h dt from the embedding vectors of y t .”
Convolutional Sequence to Sequence Learning (2017, background): “Popular choices for recurrent networks in encoder-decoder models are long short term memory networks (LSTM; Hochreiter & Schmidhuber, 1997) and gated recurrent units (GRU; Cho et al., 2014).”
Massive Exploration of Neural Machine Translation Architectures (2017, background): “In our experiments, LSTM cells consistently outperformed GRU cells .”
A Structured Self-attentive Sentence Embedding (2017, neutral): “Several models have been proposed along this line, by using recurrent networks (Hochreiter & Schmidhuber, 1997; Chung et al., 2014), recursive networks (Socher et al., 2013) and convolutional networks (Kalchbren-ner et al., 2014; dos Santos & Gatti, 2014; Kim, 2014) as an intermediate step in…”
Factorization tricks for LSTM networks (2017, methodology): “To address this issue, the LSTM cell has been introduced by Hochreiter & Schmidhuber (1997), with the following recurrent computations: LSTM : (1) where x t is input, h t is cell’s state, and c t is cell’s memory.”
Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (2017, methodology): “Previous State-of-the-Art: The best previously published results (Jozefowicz et al., 2016) use models consisting of one or more stacked Long Short-Term Memory (LSTM) layers (Hochreiter & Schmidhuber, 1997; Gers et al., 2000).”
Neural Machine Translation in Linear Time (2016, background): “Recurrent neural networks (RNN) are powerful sequence models (Hochreiter & Schmidhuber, 1997) and are widely used in language modelling (Mikolov et al., 2010), yet they have a potential drawback.”
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation (2016, neutral): “Using a human side-by-side evaluation on a set of isolated simple sentences, it reduces translation errors by an average of 60% compared to Google’s phrase-based production system.”
Using the Output Embedding to Improve Language Models (2016, methodology): “The ﬁrst model that implemented language modeling with LSTMs (Hochreiter and Schmidhuber, 1997) was (Sundermeyer et al., 2012).”
Deep Recurrent Models with Fast-Forward Connections for Neural Machine Translation (2016, methodology): “We attribute this problem to the properties of the Long ShortTerm Memory (LSTM) (Hochreiter and Schmidhuber, 1997) which is widely used in NMT.”
Exploring the Limits of Language Modeling (2016, methodology): “To this extent, we used the Long-Short Term Memory model (Hochreiter & Schmidhuber, 1997) which uses a gating mechanism (Gers et al., 2000) to ensure proper propagation of information through many time steps.”
Long Short-Term Memory-Networks for Machine Reading (2016, background): “, 1994), leading to models with gated activation functions (Hochreiter and Schmidhuber, 1997; Cho et al., 2014), and more advanced architectures that enhance the information flow within the network (Koutnı́k et al.”
Multi-task Sequence to Sequence Learning (2015, background): “These work, however, differ in terms of: (a) architecture – from unidirectional, to bidirectional, and deep multi-layer RNNs; and (b) RNN type – which are long-short term memory (LSTM) (Hochreiter & Schmidhuber, 1997) and the gated recurrent unit (Cho et al., 2014).”
Grammar as a Foreign Language (2014, methodology): “In this work, we use the same typof recurrent neural net, called Long Short-Term Memory (Hochreiter & Schmidhuber, 1997, LSTM).”
Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling (2014, background): “ a simple element-wise nonlinearity by using gating units. The earliest attempt in this direction resulted in an activation function, or a recurrent unit, called a long short-term memory (LSTM) unit [Hochreiter and Schmidhuber, 1997]. More recently, another type of recurrent unit, to which we refer ”
Sequence to Sequence Learning with Neural Networks (2014, methodology): “While it could work in principle since the RNN is provided with all the relevant information, it would be difficult to train the RNNs due to the resulting long term dependencies (figure 1) [14, 4, 16, 15].”