Gradient Flow in Recurrent Nets: The Difficulty of Learning LongTerm Dependencies

By John F. Kolen, S. C. Kremer. 2001

329 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Time-series forecasting with deep learning: a survey (2020, background): “Long short-term memory Due to the infinite lookback window, older variants of RNNs can suffer from limitations in learning long-range dependencies in the data [42,43]—due to issues with exploding and vanishing gradients [31].”
Random error sampling-based recurrent neural network architecture optimization (2019, background): “The main issue with gradient-based approaches is that they keep a vector of activations, which makes RNNs extremely deep and aggravates the exploding and the vanishing gradient problems [4, 5, 14].”
Audio-based multimedia event detection using deep recurrent neural networks (2016, background): “Even though in theory (bidirectional) RNNs can make use of information in the distant past (and future), in practice this is hard to achieve due to a problem called vanishing or exploding gradient [19].”
Domain-Independent Dominance of Adaptive Methods (2019, background): “Deep neural networks are notoriously difficult and costly to train due to the non-convexity of the underlying objective coupled with limitations of first-order methods, like vanishing and shattered gradients [15, 16, 2].”
A review on the long short-term memory model (2020, background): “Recurrent or very deep neural networks are difﬁcult to train, as they often suffer from the exploding/vanishing gradient problem (Hochreiter 1991; Kolen and Kremer 2001).”
A Survey on Deep Learning for Human Mobility (2020, background): “RNNs suffer from the vanishing gradient problem [102] and cannot propagate information found at early steps, losing relevant information at the beginning of a sequence when it is time to analyze its end [102].”
Low-Cost Recurrent Neural Network Expected Performance Evaluation (2018, methodology): “, using a local numerical optimization such as stochastic gradient descent or second order methods, due to the exploding and the vanishing gradient problems [5, 15, 18].”
Two streams Recurrent Neural Networks for Large-Scale Continuous Gesture Recognition (2016, background): “However, it has the disadvantages of the vanishing gradient and error blowing up [28].”
A novel time–frequency Transformer based on self–attention mechanism and its application in fault diagnosis of rolling bearings (2021, background): “In addition, RNN variants still cannot completely avoid long-range dependence problem, that is, the di ﬃ culty in establishing an e ﬀ ective connection between distant sequences [21].”
Social signal classification using deep blstm recurrent neural networks (2014, background): “However, standard RNNs tend to suffer from the vanishing gradient problem [17], thus limiting their access to long time lags.”
Deep Learning for Single Image Super-Resolution: A Brief Review (2018, background): “Despite remarkable progress achieved by ANNs during that period, there were still many deficiencies handicapping ANNs from developing further [37], [38].”
A Specialized Evolutionary Strategy Using Mean Absolute Error Random Sampling to Design Recurrent Neural Networks (2019, background): “The main issue with gradient-based approaches is that they keep a vector of activations, which makes RNNs extremely deep and aggravates the exploding and the vanishing gradient problems [4], [14], [5].”
On the Selection of Initialization and Activation Function for Deep Neural Networks (2018, background): “However, these functions suffer from the vanishing gradient problem during back-propagation; see, e.g., Pascanu et al. (2013) and Kolen & Kremer (2001).”