Recurrent Neural Network Regularization

By Wojciech Zaremba, Ilya Sutskever, Oriol Vinyals. 2014

3,024 citations recorded by the source corpus.

We present a simple regularization technique for Recurrent Neural Networks (RNNs) with Long Short-Term Memory (LSTM) units. Dropout, the most successful technique for regularizing neural networks, does not work well with RNNs and LSTMs. In this paper, we show how to correctly apply dropout to LSTMs, and show that it substantially reduces overfitting on a variety of tasks. These tasks include language modeling, speech recognition, image caption generation, and machine translation.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

In Signal Processing

Citations in the corpus

Factorization tricks for LSTM networks (2017, methodology): “We consider LSTM cell with projection of size p , LSTMP, where Equation 1 is computed as follows (Sak et al., 2014; Zaremba et al., 2014).”
Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (2017, methodology): “For every layer other than the softmax, we apply drouput (Zaremba et al., 2014) to the layer output, dropping each activation with probability DropProb , otherwise dividing by (1 − DropProb ) .”
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation (2016, methodology): “To prevent overﬁtting, we apply dropout during training with a scheme similar to [44].”
Exploring the Limits of Language Modeling (2016, methodology): “Much work has been done on small and large scale RNN-based LMs (Mikolov et al., 2010; Mikolov, 2012; Chelba et al., 2013; Zaremba et al., 2014; Williams et al., 2015; Ji et al., 2015a; Wang & Cho, 2015; Ji et al., 2015b).”
Recurrent Neural Network Grammars (2016, methodology): “In the last few years, language model results that substantially improve over long-established state-ofthe-art baselines have been obtained using RNNs (Zaremba et al., 2015; Mikolov et al., 2010) as well as in various conditional language modeling tasks such as machine translation (Bahdanau et al.”
Effective Approaches to Attention-based Neural Machine Translation (2015, methodology): “We use the LSTM unit defined in (Zaremba et al., 2015).”
Character-Level Language Modeling with Deeper Self-Attention (2018, methodology): “As with this work, progress has been made with discovering ways to regularize sequential architectures, with techniques such as Recurrent Dropout [48, 11] and Zoneout [26, 36].”
Semi-supervised Sequence Learning (2015, methodology): “With random embedding dimension dropout [38] and random word dropout (not published previously), we are able to reach performance of around 86.”
Transformer-XL: Attentive Language Models beyond a Fixed-Length Context (2019, background): “…2016; Chung et al., 2016; Merity et al., 2016; Kalchbrenner et al., 2016; Al-Rfou et al., 2018), improving regularization and optimization algorithms Zaremba et al. (2014); Inan et al. (2016); Press & Wolf (2016); Merity et al. (2017); Gal & Ghahramani (2016), speeding up the Softmax computation…”
Matching Networks for One Shot Learning (2016, neutral): “Future work should explore combining parametric models such as an LSTM-LM with non-parametric components such as the Matching Networks explored here.”
Unsupervised Pretraining for Sequence to Sequence Learning (2016, methodology): “We used a learning rate of 5e-5 which is multiplied by 0.8 every 50K steps after an initial 400K steps, gradient clipping with norm 5.0 (Pascanu et al., 2013), and dropout of 0.2 on non-recurrent connections (Zaremba et al., 2014).”
Pointer Sentinel Mixture Models (2016, methodology): “Our experimental setup reﬂects that of Zaremba et al. (2014) and Gal (2015).”
Language Models as Knowledge Bases? (2019, background): “…representations that could transfer factual and commonsense knowledge reliably to downstream tasks such as commonsense (visual) question answering (Zellers et al., 2018; Talmor et al., 2019) or reinforcement learning (Branavan et al., 2011; Chevalier-Boisvert et al., 2018; Bahdanau et al., 2019;…”
Improving Neural Language Models with a Continuous Cache (2016, positive): “We train recurrent neural network language models with 1024 LSTM units, regularized with dropout (probability of dropping out units equals to 0 .”
Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models (2014, background): “Dropout [34] strategies have also been proposed to prevent overﬁtting in deep LSTMs. [35] Let X t denote a matrix of training instances at time t .”
Order Matters: Sequence to sequence for sets (2015, neutral): “…(Vinyals et al., 2015b), and models for computation map from problem statements (e.g. a python program or a set of points on the plane) to their solutions (the answer to the program (Zaremba & Sutskever, 2014), or the traveling salesman problem tour for the set of points (Vinyals et al., 2015a)).”
Ethical Challenges in Data-Driven Dialogue Systems (2017, methodology): “We use Word2Vec embeddings and the corresponding debiased versions from (Bolukbasi et al. 2016) to train two LSTM language models as in (Zaremba, Sutskever, and Vinyals 2014) over Google one-billion words benchmark dataset (Chelba et al. 2013).”