Learning long-term dependencies with gradient descent is difficult

By Yoshua Bengio, Patrice Y. Simard, Paolo Frasconi. 1994

Subfields: Model Architectures, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

In Scaling & Learning Theory

Citations in the corpus

Deep Residual Learning for Image Recognition (2015, background): “Driven by the significance of depth, a question arises: Is learning better networks as easy as stacking more layers? An obstacle to answering this question was the notorious problem of vanishing/exploding gradients [1, 9], which hamper convergence from the beginning.”
Factorization tricks for LSTM networks (2017, background): “Learning long-range dependencies with Recurrent Neural Networks (RNN) is challenging due to the vanishing and exploding gradient problems (Bengio et al., 1994; Pascanu et al., 2013).”
Deep Recurrent Models with Fast-Forward Connections for Neural Machine Translation (2016, methodology): “When the model is trained with larger depth without F-F connections, we find that the parameter exploding problem (Bengio et al., 1994) happens so frequently that we could not finish training.”
Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling (2014, background): “These shortcuts allow the error to be back-propagated easily without too quickly vanishing (if the gating unit is nearly saturated at 1 ) as a result of passing through multiple, bounded nonlinearities, thus reducing the difﬁculty due to vanishing gradients [Hochreiter, 1991, Bengio et al., 1994].”
Sequence to Sequence Learning with Neural Networks (2014, background): “While it could work in principle since the RNN is provided with all the relevant information, it would be difficult to train the RNNs due to the resulting long term dependencies (figure 1) [14, 4, 16, 15].”
Neural Machine Translation by Jointly Learning to Align and Translate (2014, background): “These paths allow gradients to ﬂow backward easily without suffering too much from the vanishing effect (Hochreiter, 1991; Bengio et al. , 1994; Pascanu et al. , 2013a).”
Generating Sequences With Recurrent Neural Networks (2013, background): “These make it easier to train deep networks, by reducing the number of processing steps between the bottom of the network and the top, and thereby mitigating the ‘vanishing gradient’ problem [1].”
RL$^2$: Fast Reinforcement Learning via Slow Reinforcement Learning (2016, methodology): “To alleviate the difﬁculty of training RNNs due to vanishing and exploding gradients (Bengio et al., 1994), we use Gated Recurrent Units (GRUs) (Cho et al., 2014) which have been demonstrated to have good empirical performance (Chung et al., 2014; J´ozefowicz et al., 2015).”
Inferring Algorithmic Patterns with Stack-Augmented Recurrent Nets (2015, methodology): “Note that if we would use linear hidden units in RNN, it would be in theory possible to make them work on these sequences, but learning a linear RNN is quite challenging because of the exploding gradient problem (Bengio et al., 1994).”
ShapeWorld - A new test methodology for multimodal language understanding (2017, background): “This kind of task is particularly common for neural network models (see, for instance, Bengio et al. (1994) more than twenty years ago).”
Deep Networks with Stochastic Depth (2016, background): “Vanishing Gradients is a well known nuisance in neural networks with many layers [11].”
Realtime Multi-person 2D Pose Estimation Using Part Affinity Fields (2016, background): “Intermediate supervisions are enforced at the end of each stage to address the problem of vanishing gradients [14, 12, 5] during training.”
Convolutional Pose Machines (2016, background): “The underlying assumption of these models is that larger parts (that correspond to full limbs instead of joints) can often have discriminative image structure that can be easier to detect and consequently help reason about the location of smaller, harder-to-detect parts.”
The Reversible Residual Network: Backpropagation Without Storing Activations (2017, background): “One of the main difficulties in training very deep networks is the problem of exploding and vanishing gradients, first observed in the context of recurrent neural networks [3].”
On the difficulty of training recurrent neural networks (2012, background): “In what follows we will try to formalize these intuitions (extending a similar derivation done in Bengio et al. (1994) where only a single hidden unit case was considered).”
Light Gated Recurrent Units for Speech Recognition (2018, background): “however, can be complicated by vanishing and exploding gradients, that might impair learning long-term dependencies [44].”
The Goldilocks Principle: Reading Children's Books with Explicit Memory Representations (2015, background): “This may be a symptom of the difﬁculty of storing and retaining information across large numbers of time steps that has been previously observed in recurrent networks (See e.g. Bengio et al. (1994)).”