Advances in optimizing recurrent networks

By Yoshua Bengio, Nicolas Boulanger-Lewandowski, Razvan Pascanu. 2012

538 citations recorded by the source corpus.

After a more than decade-long period of relatively little research activity in the area of recurrent neural networks, several new developments will be reviewed here that have allowed substantial progress both in understanding and in technical solutions towards more efficient training of recurrent networks. These advances have been motivated by and related to the optimization issues surrounding deep learning. Although recurrent networks are extremely powerful in what they can in principle represent in terms of modeling sequences, their training is plagued by two aspects of the same issue regarding the learning of long-term dependencies. Experiments reported here evaluate the use of clipping gradients, spanning longer time ranges with leaky integration, advanced momentum techniques, using more powerful output probability models, and encouraging sparser gradients to help symmetry breaking and credit assignment. The experiments are performed on text and music data and show off the combined effects of these techniques in generally improving both training and test error.

Subfields: Artificial Intelligence, Optimization & Training, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Optimization & Training

In Signal Processing

Citations in the corpus

Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling (2014, methodology): “One such approach is to devise a better learning algorithm than a simple stochastic gradient descent [see, e.g., Bengio et al., 2013, Pascanu et al., 2013, Martens and Sutskever, 2011], for example using the very simple clipped gradient , by which the norm of the gradient vector is clipped, or…”
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation (2014, background): “Furthermore, this may be considered an adaptive variant of a leaky-integration unit (Bengio et al., 2013).”
Learning to learn by gradient descent by gradient descent (2016, background): “These global averaging cells (GACs) are sufﬁcient to allow the networks to implement L2 gradient clipping [Bengio et al., 2013] assuming that each LSTM can compute the square of the gradient.”
Learning to Execute (2014, methodology): “…to use Recurrent Neural Networks with Long Short Term Memory units (Hochreiter & Schmidhuber, 1997), although there are many other RNN variants that perform well on tasks with long term dependencies (Cho et al., 2014; Jaeger et al., 2007; Koutn´ık et al., 2014; Martens, 2010; Bengio et al., 2013).”
On the difficulty of training recurrent neural networks (2012, methodology): “SGD-CR provides a statistically signiﬁcant improvement on the state-of-the-art for RNNs on all the polyphonic music prediction tasks except for MuseData on which we get exactly the same performance as the state-of-the-art (Bengio et al. , 2012), which uses a diﬀerent architecture.”
Dueling Network Architectures for Deep Reinforcement Learning (2015, background): “This clipping is not standard practice in deep RL, but common in recurrent network training (Bengio et al., 2013).”
On the Variance of the Adaptive Learning Rate and Beyond (2019, methodology): “Thus, the proposed rectiﬁcation term is orthogonal to other training stabilization techniques such as gradient clipping (Bengio et al., 2013), smoothing the adaptive learning rate ( i.e. , increasing (cid:15) , applying geometric mean ﬁlter (Chen et al., 2018), or adding range constraints (Luo et…”
Learning Longer Memory in Recurrent Neural Networks (2014, positive): “n somewhat unfair - with the input, forget and output gates, the LSTM has about 4x more parameters than SRN with the same size of hidden layer. Comparison to ”leaky neurons” is also in favor of SCRN: Bengio et al. (2013) report perplexity reduction from 136 (SRN) to 129 (SRN + leaky neurons), while ”
Recurrent Latent Variable Networks for Session-Based Recommendation (2017, background): “Specifically, the introduction of novel treatments of Recurrent Neural Networks (RNNs) [4], with compelling performance in as challenging and diverse tasks as image recognition, natural language understanding, and video captioning,”
Deep Learning: Methods and Applications (2014, background): “Many deep networks in this category can be used to meaningfully generate samples by sampling from the networks, with examples of RBMs, DBNs, DBMs, and generalized denoising autoencoders [23], and are thus generative models.”
Spectral Norm Regularization for Improving the Generalizability of Deep Learning (2017, methodology): “We trained models using Nesterov’s accelerated gradient descent [2] with momentum 0.9.”
An overview of gradient descent optimization algorithms (2016, background): “This anticipatory update prevents us from going too fast and results in increased responsiveness, which has signiﬁcantly increased the performance of RNNs on a number of tasks [2].”
Deep Learning of Representations: Looking Forward (2013, background): “An even more radically different solution to the problem of avoiding explicit latent variables was recently introduced in Bengio et al. (2013c) and Bengio and Thibodeau-Laufer (2013).”
Learning Stochastic Recurrent Networks (2014, methodology): “This is partially due to architectural enhancements [6], new optimisation ﬁndings [7, 8, 9] and the increased computional power available to researchers.”
Data-Driven Sparse Structure Selection for Deep Neural Networks (2017, methodology): “For optimization, we adopt NAG [38, 3] and our modified APG to update weights W and scaling factors λ, respectively.”
Deep Sentence Embedding Using Long Short-Term Memory Networks: Analysis and Application to Information Retrieval (2015, methodology): “The basic version of RNN The RNN is a type of deep neural networks that are “deep” in temporal dimension and it has been used extensively in time sequence modelling [21], [22], [23], [24], [25], [26], [27], [28], [29].”
BlackOut: Speeding up Recurrent Neural Network Language Models With Very Large Vocabularies (2015, methodology): “We use RMSProp (Hinton, 2012) for learning rate scheduling and gradient clipping (Bengio et al., 2013) to avoid the gradient explosion issue of recurrent networks.”