Regularizing RNNs by Stabilizing Activations

By David Krueger, Roland Memisevic. 2015

82 citations recorded by the source corpus.

We stabilize the activations of Recurrent Neural Networks (RNNs) by penalizing the squared distance between successive hidden states' norms. This penalty term is an effective regularizer for RNNs including LSTMs and IRNNs, improving performance on character-level language modeling and phoneme recognition, and outperforming weight noise and dropout. We achieve competitive performance (18.6\% PER) on the TIMIT phoneme recognition task for RNNs evaluated without beam search or an RNN transducer. With this penalty term, IRNN can achieve similar performance to LSTM on language modeling, although adding the penalty term to the LSTM results in superior performance. Our penalty term also prevents the exponential growth of IRNN's activations outside of their training horizon, allowing them to generalize to much longer sequences.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Data Noising as Smoothing in Neural Network Language Models (2017, methodology): “Others have introduced methods for recurrent neural networks encouraging the hidden activations to remain stable in norm, or constraining the recurrent weight matrix to have eigenvalues close to one (Krueger & Memisevic, 2015; Arjovsky et al., 2015; Le et al., 2015).”
An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling (2018, methodology): “On character-level language modeling (PTB and text8, accuracy measured in bits per character), the generic TCN outperforms regularized LSTMs and GRUs as well as methods such as Norm-stabilized LSTMs (Krueger & Memisevic, 2015).”
On Multiplicative Integration with Recurrent Neural Networks (2016, methodology): “48, which is comparable to a vanilla-RNN with stabilizing regularizer [14], while lin-RNN performs rather poorly, achieving a test BPC of over 2.”
Architectural Complexity Measures of Recurrent Neural Networks (2016, neutral): “Deﬁnition 3.3.1.”
On orthogonality and learning recurrent networks with long term dependencies (2017, background): “Krueger & Memisevic (2015) stabilize the norm of propagating signals directly by penalizing differences in successive norm pairs in the forward pass and Pascanu et al. (2013) propose to penalize successive gradient norm pairs in the backward pass.”
Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNN (2018, background): “In [24], a penalty term on the squared distance between successive hidden states’ norms was proposed to prevent the exponential growth of IRNN’s activation.”
Sparsely Changing Latent States for Prediction and Planning in Partially Observable Domains (2021, background): “Loss-based regularization of latent updates: For latent state regularization, Krueger and Memisevic [32] have proposed using an auxiliary loss term that punishes the change in L2-norms of the latent state, which results in piecewise constant norms but not dynamics of the hidden states.”
Hierarchical Multiscale Recurrent Neural Networks (2016, unknown): “Penn Treebank Model BPC Norm-stabilized RNN (Krueger & Memisevic, 2015) 1.48 CW-RNN (Koutník et al., 2014) 1.46 HF-MRNN (Mikolov et al., 2012) 1.41 MI-RNN (Wu et al., 2016) 1.39 ME n -gram (Mikolov et al., 2012) 1.37 BatchNorm LSTM (Cooijmans et al., 2016) 1.32 Zoneout RNN (Krueger et al., 2016) 1.2”
Text Generation with Exemplar-based Adaptive Decoding (2019, unknown): “As pointed out by Krueger and Memisevic (2017), the parameter matrices of a trained RNN tend to have full rank.”
Recurrent Batch Normalization (2016, unknown): “td -LSTM (Zhang et al., 2016) 1.63 HF-MRNN (Mikolov et al., 2012) 1.54 skipping RNN (Pachitariu & Sahani, 2013) 1.48 LSTM (ours) 1.43 BN-LSTM (ours) 1.36 HM-LSTM (Chung et al., 2016) 1.29 Table 3: Bits-per-character on the text8 test sequence.”
Feed-Forward Networks with Attention Can Solve Some Long-Term Memory Problems (2015, methodology): “However, these approaches don’t solve the problem of vanishing and exploding gradients, and as a result RNNs are in practice typically only applied in tasks where sequential dependencies span at most hundreds of time steps [6, 7, 11, 3, 13, 14].”
Revisiting Activation Regularization for Language RNNs (2017, background): “Norm stabilization (Krueger & Memisevic, 2015) penalizes the model when the norm of an RNN’s hidden state changes substantially between timesteps, achieving strong results in character language modeling on and phoneme recognition.”
Twin Networks: Matching the Future for Sequence Generation (2017, methodology): “Krueger & Memisevic (2015) introduce a “norm stabilization” regularization term that ensures that the consecutive hidden states of an RNN have similar Euclidean norm.”
Recurrent Orthogonal Networks and Long-Memory Tasks (2016, methodology): “Two recent methods propose to directly address the vanishing gradient problem by either initializing or parameterizing the transition matrix with orthogonal or unitary matrices (Arjovsky et al., 2015; Le et al., 2015).”
Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations (2016, unknown): “075), and norm stabilizer (β = 50) [Krueger and Memisevic, 2015].”
Orthogonal RNNs and Long-Memory Tasks (2016, methodology): “Other methods have been proposed to deal with learning long-term dependencies, such as adding a separate contextual memory (Tomas Mikolov, 2015), stabilizing activations (Krueger & Mimesevic, 2015) or us-ing more sophisticated optimization schemes (Martens & Sutskever, 2011).”
Recurrent Neural Networks With Limited Numerical Precision (2016, background): “In [Krueger and Memisevic, 2015] it was shown that in a trained model, hidden state norms change in the first several timesteps, but become stable afterwards.”