On the difficulty of training recurrent neural networks
By Razvan Pascanu, Tomas Mikolov, Yoshua Bengio. 2012
6,009 citations recorded by the source corpus.
There are two widely known issues with properly training recurrent neural networks, the vanishing and the exploding gradient problems detailed in Bengio et al. (1994). In this paper we attempt to improve the understanding of the underlying issues by exploring these problems from an analytical, a geometric and a dynamical systems perspective. Our analysis is used to justify a simple yet effective solution. We propose a gradient norm clipping strategy to deal with exploding gradients and a soft constraint for the vanishing gradients problem. We validate empirically our hypothesis and proposed solutions in the experimental section.
Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Model Architectures
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Mohit Bansal
- Jianfeng Gao
- Quoc V. Le
- Percy Liang
- Luke S. Zettlemoyer
- Sergey Levine
- Kyunghyun Cho
In Statistical and Nonlinear Physics
- Yoshua Bengio
- Arash Vahdat
- Stefano Ermon
- Greg Yang
- Aaron C. Courville
- Kyunghyun Cho
- Sam Schoenholz
- Yang Song
In Model Architectures
- Yoshua Bengio
- Quoc V. Le
- Kyunghyun Cho
- Ruslan Salakhutdinov
- Richard Socher
- Mohit Bansal
- Vladlen Koltun
- Yoav Goldberg
Citations in the corpus
- Convolutional Sequence to Sequence Learning (2017, neutral): “1 (Pascanu et al., 2013).”
- Factorization tricks for LSTM networks (2017, background): “Learning long-range dependencies with Recurrent Neural Networks (RNN) is challenging due to the vanishing and exploding gradient problems (Bengio et al., 1994; Pascanu et al., 2013).”
- Exploring the Limits of Language Modeling (2016, methodology): “For this purpose, we will release the model weights and recipes upon publication.”
- Long Short-Term Memory-Networks for Machine Reading (2016, background): “, 1994), which can be ameliorated with gated activation functions, such as the Long Short-Term Memory (LSTM) (Hochreiter and Schmidhuber, 1997), and the gradient clipping trick (Pascanu et al., 2012).”
- Rethinking the Inception Architecture for Computer Vision (2015, background): “In addition, gradient clipping [14] with threshold 2.”
- Neural Machine Translation of Rare Words with Subword Units (2015, methodology): “We perform two independent training runs for each models, once with cut-off for gradient clipping (Pascanu et al., 2013) of 5.0, once with a cut-off of 1.0 – the latter produced better single models for most settings.”
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling (2014, methodology): “At every update, we rescale the norm of the gradient to 1, if it is larger than 1 [Pascanu et al., 2013] to prevent exploding gradients.”
- Sequence to Sequence Learning with Neural Networks (2014, background): “Thus we enforced a hard constraint on the norm of the gradient [10, 25] by scaling it when its norm exceeded a threshold.”
- Neural Machine Translation by Jointly Learning to Align and Translate (2014, methodology): “…hidden state s 0 is computed by s The context vector c i are recomputed at each step by the alignment model: normalized the L 2 -norm of the gradient of the cost function each time to be at most a predefined threshold of 1 , when the norm was larger than the threshold (Pascanu et al. , 2013b).”
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift (2015, methodology): “Our future work includes applications of our method to Recurrent Neural Networks (Pascanu et al., 2013), where the internal covariate shift and the vanishing or exploding gradients may be especially severe, and which would allow us to more thoroughly test the hypothesis that normalization improves…”
- Deeply-Supervised Nets (2014, background): “In addition to the present observation that learned features in CNN are not always intuitive and discriminative [29], broader discussion of difficulties in training deep neural networks has appeared in [8, 19].”
- Exact solutions to the nonlinear dynamics of learning in deep linear neural networks (2013, background): “Finally, we note that in recurrent networks, which can be thought of as infinitely deep feed-forward networks with tied weights, a very promising approach is a modification to the training objective that partially promotes dynamical isometry for the set of gradients currently being back-propagated [”
- Fast large-scale optimization by unifying stochastic gradient and quasi-Newton methods (2013, background): “Quadratic functions are often a poor match to the geometry of the objective function (Pascanu et al., 2012).”
- Cloze-driven Pretraining of Self-attention Networks (2019, neutral): “1 (Pascanu et al., 2013).”
- Unsupervised Pretraining for Sequence to Sequence Learning (2016, methodology): “0 (Pascanu et al., 2013), and dropout of 0.”
- Adaptive Input Representations for Neural Language Modeling (2018, neutral): “1 (Pascanu et al., 2013).”
- Character-level Convolutional Networks for Text Classification (2015, methodology): “We also used gradient clipping [25] in which the gradient norm is limited to 5.”
- Pointer Sentinel Mixture Models (2016, background): “The gradients are rescaled if their global norm exceeds 1 (Pascanu et al., 2013b).3 We evaluate the medium model configuration which features a hidden size of H = 650 and a two layer LSTM.”