Revisiting Activation Regularization for Language RNNs

By Stephen Merity, Bryan McCann, Richard Socher. 2017

45 citations recorded by the source corpus.

Recurrent neural networks (RNNs) serve as a fundamental building block for many sequence tasks across natural language processing. Recent research has focused on recurrent dropout techniques or custom RNN cells in order to improve performance. Both of these can require substantial modifications to the machine learning model or to the underlying RNN configurations. We revisit traditional regularization techniques, specifically L2 regularization on RNN activations and slowness regularization over successive hidden states, to improve the performance of RNNs on the task of language modeling. Both of these techniques require minimal modification to existing RNN architectures and result in performance improvements comparable or superior to more complicated regularization techniques or custom cell architectures. These regularization techniques can be used without any modification on optimized LSTM implementations such as the NVIDIA cuDNN LSTM.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Improving Neural Language Modeling via Adversarial Training (2019, methodology): “The WT2 and WT103 datasets are introduced in Merity et al. (2017b) as an alternative to the PTB dataset, and which contain lightly pre-possessed Wikipedia articles.”
An Analysis of Neural Language Modeling at Multiple Scales (2018, methodology): “…are useful in many applications including speech recognition (Yu & Deng, 2014), machine translation (Koehn, 2009), natural language generation (Radford et al., 2017; Merity et al., 2017a), learning token embeddings, and as a general-purpose feature extractor for downstream tasks (P. Matthew, 2017).”
Accelerating Convolutional Neural Networks via Activation Map Compression (2018, methodology): “Finally, activation map regularization has appeared in the literature in various forms such as dropout [57], batch normalization [27], layer normalization [4] and L 2 regularization [41].”
Fraternal Dropout (2017, methodology): “The authors of Merity et al. (2017b) study the importance of activity regularization (AR) 8 and temporal activity regularization (TAR) in LSTMs given as, where h t ∈ R d is the LSTM’s output activation at time step t (hence depends on both current input z t and the model parameters θ ).”
The Implicit and Explicit Regularization Effects of Dropout (2020, methodology): “In NLP, dropout is the norm on both small and large models, as it is much more eﬀective than methods such as (cid:96) 2 regularization (Merity et al., 2017a).”
Regularizing and Optimizing LSTM Language Models (2017, methodology): “In addition, L 2 decay can be used on the individual unit activations and on the difference in outputs of an RNN at different time steps; these strategies labeled as activation regularization (AR) and temporal activation regularization (TAR) respectively (Merity et al., 2017).”