Gradient Flow in Recurrent Nets: the Difficulty of Learning Long-Term Dependencies

By Sepp Hochreiter, Yoshua Bengio. 2001

2,083 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

Citations in the corpus

Attention is All you Need (2017, background): “The shorter these paths between any combination of positions in the input and output sequences, the easier it is to learn long-range dependencies [11].”
Neural Machine Translation in Linear Time (2016, background): “The larger the distance, the harder it is to learn the dependencies between the tokens (Hochreiter et al., 2001).”
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation (2016, background): “too slow and difficult to train, likely due to exploding and vanishing gradient problems [33, 22].”
Sequence to Sequence Learning with Neural Networks (2014, background): “While it could work in principle since the RNN is provided with all the relevant information, it would be difficult to train the RNNs due to the resulting long term dependencies (figure 1) [14, 4, 16, 15].”
Generating Sequences With Recurrent Neural Networks (2013, background): “In practice however, standard RNNs are unable to store information about past inputs for very long [15].”
Character-Level Language Modeling with Deeper Self-Attention (2018, methodology): “Much of the progress in this area has been made by mitigating the vanishing gradients problem (Hochreiter et al. 2001) by architectures such as LSTMs (Hochreiter and Schmidhuber 1997), GRU (Cho et al.”
Semi-supervised Sequence Learning (2015, background): “Recurrent neural networks (RNNs) are powerful tools for modeling sequential data, yet training them by back-propagation through time [36, 26] can be difﬁcult [8].”
Transformer-XL: Attentive Language Models beyond a Fixed-Length Context (2019, background): “Despite the wide adaption, RNNs are difficult to optimize due to gradient vanishing and explosion (Hochreiter et al., 2001), and the introduction of gating in LSTMs and the gradient clipping technique (Graves, 2013) might not be sufficient to fully address this issue.”
Universal Transformers (2018, background): “…architectures address a significant shortcoming of RNNs, namely their inherently sequential computation which prevents parallelization across elements of the input sequence, whilst still addressing the vanishing gradients problem as the sequence length gets longer (Hochreiter et al., 2003).”
Adaptive Computation Time for Recurrent Neural Networks (2016, background): “N E ] 2 1 Fe make every network very deep and design its architecture in such a way as to mitigate the vanishing gradient problem [13] associated with long chains of iteration [29, 17].”
Neural Turing Machines (2014, background): “This very general architecture was developed for a speciﬁc purpose, to address the “vanishing and exploding gradient” problem (Hochreiter et al., 2001), which we might relabel the problem of “vanishing and exploding sensitivity.””
Convolutional Pose Machines (2016, background): “Composing multiple convolutional networks in a CPM results in an overall network with many layers that is at risk of the problem of vanishing gradients [4, 5, 10, 12] during learning.”
End-to-End Training of Deep Visuomotor Policies (2015, background): “This issue has also been observed in the related context of recurrent neural networks (Hochreiter et al., 2001; Pascanu and Bengio, 2012).”
Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs) (2015, background): “Besides producing sparse codes, the main advantage of ReLUs is that they alleviate the vanishing gradient problem (Hochreiter, 1998; Hochreiter et al., 2001) since the derivative of 1 for positive values is not contractive (Glorot et al., 2011).”
Multi-column deep neural networks for image classification (2012, neutral): “MCDNN of 5 nets trained with the same preprocessor achieve better results than their constituent DNNs, except for original images (Tab.”
Sequence Transduction with Recurrent Neural Networks (2012, background): “While the early years of RNNs were dogged by difficulties in learning (Hochreiter et al., 2001), recent results have shown that they are now capable of delivering state-of-the-art results in real-world tasks such as handwriting recognition (Graves et al., 2008; Graves & Schmidhuber, 2008), text…”
The Best of Both Worlds: Combining Recent Advances in Neural Machine Translation (2018, background): “In practice, RNNs are notoriously hard to train (Hochreiter, 1991; Bengio et al., 1994; Hochreiter et al., 2001), confirming the well”
Recurrent Highway Networks (2016, methodology): “Correspondence to: Julian Zilly”
Grid Long Short-Term Memory (2015, background): “…of the previous memory vector and the gates; this ensures that the forward signals from one step to the other are not repeatedly squashed by a non-linearity such as tanh and that the backward error signals do not decay sharply at each step, an issue known as the vanishing gradient problem [19].”