Regularization and nonlinearities for neural language models: when are they needed?

By Marius Pachitariu, Maneesh Sahani. 2013

47 citations recorded by the source corpus.

Neural language models (LMs) based on recurrent neural networks (RNN) are some of the most successful word and character-level LMs. Why do they work so well, in particular better than linear neural LMs? Possible explanations are that RNNs have an implicitly better regularization or that RNNs have a higher capacity for storing patterns due to their nonlinearities or both. Here we argue for the first explanation in the limit of little training data and the second explanation for large amounts of text data. We show state-of-the-art performance on the popular and small Penn dataset when RNN LMs are regularized with random dropout. Nonetheless, we show even better performance from a simplified, much less expressive linear RNN model without off-diagonal entries in the recurrent matrix. We call this model an impulse-response LM (IRLM). Using random dropout, column normalization and annealed learning rates, IRLMs develop neurons that keep a memory of up to 50 words in the past and achieve a perplexity of 102.5 on the Penn dataset. On two large datasets however, the same regularization methods are unsuccessful for both models and the RNN's expressivity allows it to overtake the IRLM by 10 and 20 percent perplexity, respectively. Despite the perplexity gap, IRLMs still outperform RNNs on the Microsoft Research Sentence Completion (MRSC) task. We develop a slightly modified IRLM that separates long-context units (LCUs) from short-context units and show that the LCUs alone achieve a state-of-the-art performance on the MRSC task of 60.8%. Our analysis indicates that a fruitful direction of research for neural LMs lies in developing more accessible internal representations, and suggests an optimization regime of very high momentum terms for effectively training such models.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

A Theoretically Grounded Application of Dropout in Recurrent Neural Networks (2015, methodology): “The model’s input is a sequence of symbols, where at each time step a simple neural network ( RNN unit ) is applied to a single symbol, as well as to the network’s output from the previous time step.”
Higher Order Recurrent Neural Networks (2016, methodology): “We have used the weight decay, momentum and column normalization (Pachitariu & Sahani, 2013) in our experiments to improve model generalization.”
Regularizing RNNs by Stabilizing Activations (2015, background): “Other regularizers for RNNs that do not target norm stability include weight noise (Jim et al., 1996) and dropout (Pham et al., 2013; Pachitariu & Sahani, 2013; Zaremba et al., 2014).”
Pushing the bounds of dropout (2018, background): “In related works, Pachitariu & Sahani (2013) in their investigation of regularisation of standard RNN based language models dismiss applying dropout to recurrent connections “ to avoid introducing instabilities into the recurrent part of the LMs ”.”
Hierarchical Multiscale Recurrent Neural Networks (2016, unknown): “…paper at ICLR 2017 Text8 Model BPC td -LSTM (Zhang et al., 2016) 1.63 HF-MRNN (Mikolov et al., 2012) 1.54 MI-RNN (Wu et al., 2016) 1.52 Skipping-RNN (Pachitariu & Sahani, 2013) 1.48 MI-LSTM (Wu et al., 2016) 1.44 BatchNorm LSTM (Cooijmans et al., 2016) Ba, 2014) with an initial learning rate of 0 .”
Implicit Regularization in Deep Learning (2017, result): “In experiment with different network sizes, we train a two layer perceptron with ReLU activation and varying number of hidden units without Batch Normalization or dropout.”
Learning Longer Memory in Recurrent Neural Networks (2014, unknown): “This type of architecture can potentially retain information about longer term statistics, such as the topic of a text, but it does not scale well to larger datasets (Pachitariu & Sahani, 2013).”
Deep learning for short-term traffic flow prediction (2016, background): “In Polson and Sokolov [2014] the authors develop a hierarchical Bayesian model for tracking traffic flows and estimate uncertainty about state variables at any given point in time.”
Recurrent Neural Network Regularization (2014, background): “Other work that applied dropout to LSTMs is (Pachitariu & Sahani, 2013).”
On Fast Dropout and its Applicability to Recurrent Networks (2013, methodology): “This is the reason why Pachitariu and Sahani (2013) only use dropout on those parts of the RNN which are not dynamic., i.e. the connections feeding from the hidden into the output layer.”
Recurrent Batch Normalization (2016, unknown): “In addition, unlike the RNN which overwrites its content at each timestep, the update of the LSTM cell is regulated by a set of gates.”
Uncertainty in Deep Learning (2016, methodology): “This is an interesting use of the ideas above; existing literature in RNNs has established that dropout cannot be applied in RNNs apart from the forward connections [Bayer et al., 2013; Bluche et al., 2015; Pachitariu and Sahani, 2013; Pham et al., 2014; Zaremba et al., 2014].”