Understanding the exploding gradient problem

By Razvan Pascanu, Tomas Mikolov, Yoshua Bengio. 2012

563 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation (2016, background): “too slow and difficult to train, likely due to exploding and vanishing gradient problems [33, 22].”
PaLM: Scaling Language Modeling with Pathways (2022, methodology): “We use global norm gradient clipping (Pascanu et al. (2012)) with a value of 1.0 for all models.”
Transformer-XL: Attentive Language Models beyond a Fixed-Length Context (2019, methodology): “Despite the wide adaption, RNNs are difﬁcult to optimize due to gradient vanishing and explosion (Hochreiter et al., 2001), and the introduction of gating in LSTMs and the gradient clipping technique (Graves, 2013; Pascanu et al., 2012) might not be sufﬁcient to fully address this issue.”
Off-Policy Actor-Critic with Shared Experience Replay (2019, methodology): “We use the V-trace distributed reinforcement learning agent (Espeholt et al., 2018) (Pascanu et al., 2012).”
Learning to Transduce with Unbounded Memory (2015, methodology): “We used gradient clipping [19], clipping all gradients above 1.”
Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations (2016, methodology): “We optimize with Adam (Kingma & Ba, 2014), clip gradients to a maximum norm of 1 (Pascanu et al., 2012), and use early stopping, again matching the settings of Cooijmans et al.”
Gradient-based Hyperparameter Optimization through Reversible Learning (2015, background): “This phenomenon is related to the exploding-gradient problem (Pascanu et al., 2012).”
Distillation as a Defense to Adversarial Perturbations Against Deep Neural Networks (2015, background): “ samples are higher, which makes attacks harder to perform, as shown in Section V. This situation is to some extent related to the exploding gradient problem sometimes encountered while training DNNs [38]: amplitudes of gradients computed during training to update DNN parameters can explode. Adaptin”
Multi-Object Representation Learning with Iterative Variational Inference (2019, methodology): “We used gradient clipping as recommended by (Pascanu et al., 2012): if the norm of global gradient exceeds 5 .”
Generalized End-to-End Loss for Speaker Verification (2017, background): “The L2-norm of gradient is clipped at 3 [17], and the gradient scale for projection node in LSTM is set to 0.”
Advances in optimizing recurrent networks (2012, methodology): “The specific form of clipping used here was proposed in [15] and is discussed there at much greater length: when the norm of the gradient vector g for a given sequence is above a threshold, the update is done in the direction threshold g ||g|| .”
Word-Based Dialog State Tracking with Recurrent Neural Networks (2014, methodology): “Gradient clipping is used to avoid the problem of exploding gradients (Pascanu et al., 2012).”
Training Deeper Neural Machine Translation Models with Transparent Attention (2018, background): “These issues have been studied in the context of capturing long range dependencies in recurrent architectures (Pascanu et al., 2012; Hochreiter et al., 2001), but resolving these deficiencies in Transformers or LSTM seq2seq models deeper than 8 layers is unfortunately under-explored (Wang et al.,…”
Architectural Complexity Measures of Recurrent Neural Networks (2016, background): “We also compare our results to the ones reported in Pascanu et al. (2012) (denoted as “P-sh”), since they used the same model architecture as the one specified by our sh.”
Learning a SAT Solver from Single-Bit Supervision (2018, neutral): “65 (Pascanu et al., 2012).”
Avoiding pathologies in very deep networks (2014, background): “Recurrent networks Bengio et al. (1994) and Pascanu et al. (2012) analyze a related problem with gradient-based learning in recurrent nets, the “exploding-gradients” problem.”
Transformer-XL: Language Modeling with Longer-Term Dependency (2018, methodology): “However, RNNs are difﬁ-cult to optimize due to gradient vanishing and explosion (Hochreiter et al., 2001), and the introduction of gating in LSTMs and the gradient clipping technique (Graves, 2013; Pascanu et al., 2012) might not be sufﬁcient to fully address this issue.”
A review of unsupervised feature learning and deep learning for time-series modeling (2014, methodology): “This results in the problem of vanishing gradients [102] and has motivated the exploration of second-order methods for deep architectures [86] and unsupervised pre-training.”
A survey of the recent architectures of deep convolutional neural networks (2019, background): “roblems, which are not caused by overfitting but instead by an increase in the depth [56], [151]. Vanishing gradient problem not only results in higher test error but also higher training error [151]–[153]. For training deep networks, the concept of multi-path or cross-layer connectivity was propose”