A Simple Way to Initialize Recurrent Networks of Rectified Linear Units

By Quoc V. Le, Navdeep Jaitly, Geoffrey E. Hinton. 2015

763 citations recorded by the source corpus.

Learning long term dependencies in recurrent networks is difficult due to vanishing and exploding gradients. To overcome this difficulty, researchers have developed sophisticated optimization techniques and network architectures. In this paper, we propose a simpler solution that use recurrent neural networks composed of rectified linear units. Key to our solution is the use of the identity matrix or its scaled version to initialize the recurrent weight matrix. We find that our solution is comparable to LSTM on our four benchmarks: two toy problems involving long-range temporal structures, a large language modeling problem and a benchmark speech recognition problem.

Subfields: Artificial Intelligence, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

Citations in the corpus

Transformer-XL: Attentive Language Models beyond a Fixed-Length Context (2019, background): “…since the ubiquitous adaption of LSTM, many efforts have been spent on relieving the vanishing gradient problem, including better initialization (Le et al., 2015), additional loss signal (Trinh et al., 2018), augmented memory structure (Ke et al., 2018) and others that modify the internal…”
Inside-Outside Net: Detecting Objects in Context with Skip Pooling and Recurrent Neural Networks (2015, methodology): “[23] recently showed that these networks are easy to train and are good at modeling long-range dependencies, if the recurrent weight matrix is initialized to the identity matrix.”
Light Gated Recurrent Units for Speech Recognition (2018, methodology): “For instance, in [51] authors replaced tanh activations with ReLU neurons in a vanilla RNN, showing the capability of this model to learn long-term dependencies when a proper orthogonal initialization is adopted.”
Multi-Scale Context Aggregation by Dilated Convolutions (2015, background): “This is a form of identity initialization, which has recently been advocated for recurrent networks (Le et al., 2015).”
Recurrent Highway Networks (2016, methodology): “Le et al. (2015) proposed to initialize R with an identity matrix and small random values on the off-diagonals.”
Recurrent Batch Normalization (2016, background): “As a result, the upper layers are continually adapting to the shifting input distribution and unable to learn effectively.”
FractalNet: Ultra-Deep Neural Networks without Residuals (2016, background): “Identity initialization also appears in the context of recurrent networks [16].”
Learning World Graphs to Accelerate Hierarchical Reinforcement Learning (2019, neutral): “optimization of deep neural networks is sensitive to weight initialization (Mishkin & Matas, 2015; Le et al., 2015), especially for a system (Co-Reyes et al.”
RETAIN: An Interpretable Predictive Model for Healthcare using Reverse Time Attention Mechanism (2016, methodology): “In the rest of this section, we will use the abstract symbol RNN to denote any recurrent neural network variants that can cope with the vanishing gradient problem [3], such as LSTM [21], GRU [9], and IRNN [26], with any depth (number of hidden layers).”
Data Noising as Smoothing in Neural Network Language Models (2017, methodology): “Existing regularization methods, however, are typically applied to weights or hidden units within the network (Srivastava et al., 2014; Le et al., 2015) instead of directly considering the input data.”
Forecasting Across Time Series Databases using Long Short-Term Memory Networks on Groups of Similar Series (2017, background): “…state st and output ot at time step t can be formally defined by the following equations: st = fθ(Uxt +Wst−1) ot = fα(V st) Generally, fθ is a non-linear function such as, e.g., tanh, or rectified linear units (ReLu; Le et al., 2015), whereas fα uses a linear function such as softmax or sigmoid.”
An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling (2018, methodology): “First introduced by Hochreiter & Schmidhuber (1997), the adding problem has been used repeatedly as a stress test for sequence models (Martens & Sutskever, 2011; Pascanu et al., 2013; Le et al., 2015; Arjovsky et al., 2016; Zhang et al., 2016).”
Dilated Recurrent Neural Networks (2017, methodology): “The problem of vanishing and exploding gradients is partially solved by LSTM and GRU memory gates; other partial solutions include gradient clipping [18], orthogonal and unitary weight optimization [2, 14, 24], and skip connections across multiple timestamps [8, 30].”
Low-Shot Learning from Imaginary Data (2018, methodology): “Inspired by [19], we initialize the weights of our hallucinator network as block diagonal identity matrices.”
Norm-preserving Orthogonal Permutation Linear Unit Activation Functions (OPLU) (2016, background): “It is a synthetic problem for testing the ability of the neural network to capture the long-term dependencies in data [19], [12], [21].”
"What Happens If..." Learning to Predict the Effect of Forces in Images (2016, methodology): “[18] show that RNNs composed of ReLUs and initialized with identity weight matrix are as powerful as standard LSTMs.”