A Theoretically Grounded Application of Dropout in Recurrent Neural Networks

By Yarin Gal, Zoubin Ghahramani. 2015

1,727 citations recorded by the source corpus.

Recurrent neural networks (RNNs) stand at the forefront of many recent developments in deep learning. Yet a major difficulty with these models is their tendency to overfit, with dropout shown to fail when applied to recurrent layers. Recent results at the intersection of Bayesian modelling and deep learning offer a Bayesian interpretation of common deep learning techniques such as dropout. This grounding of dropout in approximate Bayesian inference suggests an extension of the theoretical results, offering insights into the use of dropout with RNN models. We apply this new variational inference based dropout technique in LSTM and GRU models, assessing it on language modelling and sentiment analysis tasks. The new approach outperforms existing techniques, and to the best of our knowledge improves on the single model state-of-the-art in language modelling with the Penn Treebank (73.4 test perplexity). This extends our arsenal of variational tools in deep learning.

Subfields: Artificial Intelligence, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Optimization & Training

Citations in the corpus

SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference (2018, methodology): “, 2017; Press and Wolf, 2017), all embedding and hidden layers are set to 512, and we used recurrent dropout (Gal and Ghahramani, 2016) on the hidden states and embedding layer.”
Character-Level Language Modeling with Deeper Self-Attention (2018, methodology): “As with this work, progress has been made with discovering ways to regularize sequential architectures, with techniques such as Recurrent Dropout [48, 11] and Zoneout [26, 36].”
Deep Contextualized Word Representations (2018, methodology): “This deep LSTM uses High-way connections (Srivastava et al., 2015) between layers and variational recurrent dropout (Gal and Ghahramani, 2016).”
MaskGAN: Better Text Generation via Filling in the ______ (2018, methodology): “We ﬁrst pretrain the commonly-used variational LSTM language model with parameter dimensions common to MaskGAN following Gal & Ghahramani (2016) to a validation perplexity of 78.”
Simple and Effective Multi-Paragraph Reading Comprehension (2017, methodology): “d Jiang, 2016), although alternative architectures exist (Chen et al., 2017; Weissenborn et al., 2017). Our model follows this approach, but includes some recent advances such as variational dropout (Gal and Ghahramani, 2016) and bi-directional attention (Seo et al., 2016). Self-attention has been u”
Transformer-XL: Attentive Language Models beyond a Fixed-Length Context (2019, background): “, 2018), improving regularization and optimization algorithms (Gal and Ghahramani, 2016) , speeding up the Softmax computation (Grave et al.”
Pointer Sentinel Mixture Models (2016, methodology): “The results for this model ablation are worse than that of Gal (2015)’s variational LSTM without Monte Carlo dropout averaging.”
Learning Recurrent Span Representations for Extractive Question Answering (2016, methodology): “We couple the input and forget gates in our LSTMs, as described in Greff et al. (2016), and we use a single dropout mask to apply dropout across all LSTM time-steps as proposed by Gal & Ghahramani (2016).”
Improving Neural Language Models with a Continuous Cache (2016, positive): “We train recurrent neural network language models with 1024 LSTM units, regularized with dropout (probability of dropping out units equals to 0 .”
Light Gated Recurrent Units for Speech Recognition (2018, methodology): “Since extending standard dropout to recurrent connections hinders learning long-term dependencies, we followed the approach introduced in [60], [61], that tackles this issue by sharing the same dropout mask across all the time steps.”
From Recognition to Cognition: Visual Commonsense Reasoning (2018, neutral): “3 [21].”
Neural Architecture Search with Reinforcement Learning (2016, methodology): “Every child model has two layers, with the number of hidden units adjusted so that total number of learnable parameters approximately match the ”medium” baselines (Zaremba et al., 2014; Gal, 2015).”
Edinburgh Neural Machine Translation Systems for WMT 16 (2016, positive): “We experimented with using automatic back-translations of the monolingual News corpus as additional training data (Sennrich et al., 2016a), pervasive dropout (Gal, 2015), and target-bidirectional models.”
Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling (2016, methodology): “We train our models using stochastic gradient descent, and we use a variant of the dropout method proposed in Gal (2015).”
Regularizing and Optimizing LSTM Language Models (2017, methodology): “A variant of this, variational dropout (Gal & Ghahramani, 2016), samples a binary dropout mask only once upon the first call and then to repeatedly use that locked dropout mask for all repeated connections within the forward and backward pass.”
Recurrent Highway Networks (2016, methodology): “Variational refers to the dropout regularization from Gal [33] based on a probabilistic interpretation of the models.”
On the State of the Art of Evaluation in Neural Language Models (2017, methodology): “For the recurrent states, both architectures utilise variational dropout proposed by Gal and Ghahramani (2016) ( state dropout ).”
End-to-end Neural Coreference Resolution (2017, background): “Dropout masks are shared across timesteps to preserve long-distance information as described in Gal and Ghahramani (2016).”