The Vanishing Gradient Problem During Learning Recurrent Neural Nets and Problem Solutions

By Sepp Hochreiter. 1998

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

Citations in the corpus

Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs) (2015, background): “Besides producing sparse codes, the main advantage of ReLUs is that they alleviate the vanishing gradient problem (Hochreiter, 1998; Hochreiter et al., 2001) since the derivative of 1 for positive values is not contractive (Glorot et al., 2011).”
Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks (2015, background): “t+Uh t 1 +b): Unfortunately, a problem with RNNs with transition functions of this form is that during training, components of the gradient vector can grow or decay exponentially over long sequences (Hochreiter, 1998; Bengio et al., 1994). This problem with exploding or vanishing gradients makes it ”
Gated Feedback Recurrent Neural Networks (2015, methodology): “Here we follow the implementation provided by Zaremba et al. (2014).”
A Character-level Decoder without Explicit Segmentation for Neural Machine Translation (2016, background): “(3) Vanishing Gradient Speciﬁcally to neural machine translation, a major reason behind the wide adoption of word-level modelling is due to the difﬁculty in modelling long-term dependencies with recurrent neural networks (Bengio et al., 1994; Hochreiter, 1998).”
Learning Longer Memory in Recurrent Neural Networks (2014, neutral): “While theoretically powerful, these recurrent models were widely considered to be hard to train due to the so-called vanishing and exploding gradient problems (Hochreiter, 1998; Bengio et al., 1994).”
Dropout Improves Recurrent Neural Networks for Handwriting Recognition (2013, background): “The burden of exploding and vanishing gradient was the reason for the lack of practical applications of RNNs until recently [6], [7].”
A practical tutorial on autoencoders for nonlinear feature fusion: Taxonomy, models, software and guidelines (2018, background): “However, a proper algorithm able to train an MLP with several hidden layers was not available, due to the vanishing gradient [6] problem.”
RUDDER: Return Decomposition for Delayed Rewards (2018, background): “ell output (A331) A8.3 Long-Term Dependencies vs. Uniform Credit Assignment The LSTM network has been proposed with the aim to learn long-term dependencies in sequences which span over long intervals [55, 56, 50, 51]. However, besides extracting long-term dependencies, LSTM memory cells have another”
Deep Learning Approach for Short-Term Stock Trends Prediction Based on Two-Stream Gated Recurrent Unit Network (2018, background): “Although RNNs have been very effective in text generation and speech recognition tasks [15], Hochreiter [25] revealed a big problem of RNNs; when the gradient is passed back through many time steps, it tends to grow or vanish, they stated that it was very hard to make RNN remember the long-term…”
Weighted Transformer Network for Machine Translation (2017, background): “However, in these models, the operations needed to learn dependencies between distant positions can be difficult to learn (Hochreiter et al., 2001; Hochreiter, 1998).”
Neural Tree Indexers for Text Understanding (2016, background): “Although a great success in a variety of tasks, RNNs have limitations (Bengio et al., 1994; Hochreiter, 1998).”
Classifying Relations via Long Short Term Memory Networks along Shortest Dependency Paths (2015, methodology): “Long short term memory (LSTM) units are proposed in Hochreiter (1998) to overcome this problem. The main idea is to introduce an adaptive gating mechanism, which decides the degree to which LSTM units keep the previous state and memorize the extracted features of the current data input. Many LSTM va”
Linguistically-Informed Self-Attention for Semantic Role Labeling (2018, methodology): “We use leaky ReLU activations to avoid dead activations and vanishing gradients (Hochreiter, 1998), whereas layer normalization reduces covariate shift between layers (Ioffe and Szegedy, 2015) without requiring distinct train- and testtime operations. We then feed this representation as input to a s”
Learning values across many orders of magnitude (2016, background): “This seems a promising way to counteract undesirable characteristics of back-propagating gradients, such as vanishing or exploding gradients [Hochreiter, 1998].”
On Extractive and Abstractive Neural Document Summarization with Transformer Language Models (2020, background): “encoder and to ease gradient ﬂow into the encoder. RNNs however, are limited by their sequential nature, making them 1) difﬁcult to optimize and learn for long sequences with long range dependencies (Hochreiter 1998; Pascanu, Mikolov, and Bengio 2013), and 2) hard to parallelize on modern hardware l”
Fast and Accurate Sequence Labeling with Iterated Dilated Convolutions (2017, background): “The loss also helps reduce the vanishing gradient problem (Hochreiter, 1998) for deep architectures.”
Deep Learning Advances in Computer Vision with 3D Data (2017, background): “However, they were very difﬁcult to train [Pascanu et al. 2013b] mainly due to the Vanishing Gradient Problem [Hochreiter 1998] initially acknowledged by Hochreiter in his diploma thesis [Hochreiter 1991].”