The problem of learning long-term dependencies in recurrent networks

By Yoshua Bengio, Paolo Frasconi, Patrice Y. Simard. 1993

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

Citations in the corpus

On the difficulty of training recurrent neural networks (2012, methodology): “We can improve our understanding of the exploding gradients and vanishing gradients problems by employing a dynamical systems perspective, as it was done before in Doya (1993); Bengio et al. (1993).”
Understanding intermediate layers using linear classifier probes (2016, background): “e sure that numerical underﬂow was not the cause for the degradation. One of the popular explanation for training difﬁculties in very deep models is that of the exploding/vanishing (Hochreiter, 1991; Bengio et al., 1993). Here we would like to offer another complementary explanation, based on the ob”
Deep Information Propagation (2016, background): “This provides a formal explanation of the ‘vanishing gradients’ phenomenon that has long been observed in neural networks (Bengio et al., 1993).”
Predicting Clinical Events by Combining Static and Dynamic Information Using Recurrent Neural Networks (2016, background): “However, it was observed by Bengio et al. [15], that it is not possible for standard RNNs to capture long-term dependencies from very far in the past due to the vanishing gradient problem.”
Learning Long-Term Dependencies with (2007, methodology): “In a rst experiment we investigated the e ect of the noise variance s and of di erent initial values w0 for the self loop weight (see also [3].”
Stability and Synchronization of Discrete-Time Markovian Jumping Neural Networks With Mixed Mode-Dependent Time Delays (2009, background): “This is referred to as the information latching problem [ 2 ] that can be handled by extracting finite-state representations (also called clusters, patterns, or modes) from trained networks [3], [7], [8], [11].”
Sequential User-based Recurrent Neural Network Recommendations (2017, background): “One shortcoming of simple variants, however, is the limited number of input variables that can effectively be handled [2].”
Learning long-term dependencies with gradient descent is difficult (1994, methodology): “In a rst experiment we investigated the e ect of the noise variance s and of di erent initial values w0 for the self loop weight (see also [3].”
Random Walk Initialization for Training Very Deep Feedforward Networks (2014, background): “Since the early 90s, it has been appreciated that deep neural networks suffer from a vanishing gradient problem (Hochreiter, 1991), (Bengio et al., 1993), (Bengio et al., 1994), (Hochreiter et al., 2001).”
Computational Capabilities of Graph Neural Networks (2009, background): “It is worth noticing, however, that similar problems can be encountered also in common recurrent neural networks, e.g., when a long sequence of inputs is processed (those problems are usually referred to as long term dependencies problems [ 28 ]).”
Neural net architectures for temporal sequence processing (2007, background): “Bengio, Frasconi, and Simard (1993) present theoretical arguments for inherent limitations of learning in recurrent networks.”
Exponential stability of delayed recurrent neural networks with Markovian jumping parameters (2006, background): “Such a phenomenon is referred to as the problem of information latching [1].”
Understanding the exploding gradient problem (2012, background): “Training Recurrent Neural Networks is more troublesome than feedforward ones because of the vanishing and exploding gradient problems detailed in Bengio et al. (1994). In this paper we attempt to understand the fundamental issues underlying the exploding gradient problem by exploring it from an anal”