Recurrent Dropout without Memory Loss

By Stanislau Semeniuta, Aliaksei Severyn, Erhardt Barth. 2016

227 citations recorded by the source corpus.

This paper presents a novel approach to recurrent neural network (RNN) regularization. Differently from the widely adopted dropout method, which is applied to forward connections of feedforward architectures or RNNs, we propose to drop neurons directly in recurrent connections in a way that does not cause loss of long-term memory. Our approach is as easy to implement and apply as the regular feed-forward dropout and we demonstrate its effectiveness for the most effective modern recurrent network – Long Short-Term Memory network. Our experiments on three NLP benchmarks show consistent improvements even when combined with conventional feed-forward dropout.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

HyperNetworks (2016, methodology): “The only place we use dropout is in the single location in Equation 13, developed in Recurrent Dropout without Memory Loss (Semeniuta et al., 2016).”
Regularizing and Optimizing LSTM Language Models (2017, background): “One such approach is taken by Semeniuta et al. (2016) wherein the authors drop updates to network units, speciﬁcally the input gates of the LSTM, in lieu of the units themselves.”
On the State of the Art of Evaluation in Neural Language Models (2017, background): “1 or recurrent dropout (Semeniuta et al., 2016) where explicitly noted.”
Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations (2016, background): “We now describe several recent works [Semeniuta et al., 2016, Moon et al., 2015, Gal, 2015] that propose variants of dropout for use in RNNs. Semeniuta et al. [2016] apply recurrent dropout to the updates to LSTM memory cells (or GRU states), i.e. they drop out the input/update gate in LSTM/GRU.”
Data Noising as Smoothing in Neural Network Language Models (2017, methodology): “Classical regularization methods such as L 2 -regularization are typically applied to the model parameters, while dropout is applied to activations which can be along the forward as well as the recurrent directions (Zaremba et al., 2014; Semeniuta et al., 2016; Gal, 2015).”
Phased LSTM: Accelerating Recurrent Network Training for Long or Event-based Sequences (2016, background): “Alterna-tively, the rhythmic periodicity can be viewed as a kind of persistent dropout that preserves state [27], enhancing model diversity.”
Dilated Recurrent Neural Networks (2017, background): “Unless stated otherwise, no tricks, such as gradient clipping [18], learning rate annealing, recurrent weight dropout [20], recurrent batch norm [20], layer norm [3], etc. , are applied.”
A Neural Representation of Sketch Drawings (2017, methodology): “For the decoder RNN, we use HyperLSTM, as this type of RNN cell excels at sequence generation tasks [8].”
Flipout: Efficient Pseudo-Independent Weight Perturbations on Mini-Batches (2018, methodology): “We used the following hyperparameters for each model: • For recurrent dropout (Semeniuta et al., 2016), we used 0.25 dropout probability on the cell state, and per-step mask sampling.”
Contextualized Word Representations for Reading Comprehension (2017, methodology): “For any single sequence, the same hidden state dropout mask was used across timesteps (Moon et al., 2015; Semeniuta et al., 2016; Gal and Ghahramani, 2016).”
The RWTH/UPB/FORTH System Combination for the 4th CHiME Challenge Evaluation (2016, methodology): “Here, we sample the mask once per sequence [18].”
Sockeye: A Toolkit for Neural Machine Translation (2017, background): “One can also enable variational dropout [Gal and Ghahramani, 2016] to sample a fixed dropout mask across recurrent time steps, or recurrent dropout without memory loss [Semeniuta et al., 2016].”
Divide and Conquer: A Deep CASA Approach to Talker-Independent Monaural Speaker Separation (2019, methodology): “To address this issue, various dropout techniques [9], [30], [33] have been proposed for RNNs.”
Memory Architectures in Recurrent Neural Network Language Models (2018, methodology): “We also use recurrent dropout (Semeniuta et al., 2016) and set it to 0.”
Representation Mixing for TTS Synthesis (2018, unknown): “Our decoder LSTMs utilize cell dropout [21] as seen in prior work [31, 32].”
Recent Advances in Recurrent Neural Networks (2017, methodology): “A similar model to the RNNDrop is introduced in [83], where instead of dropout, it masks data samples at every input sequence per step.”
Multimodal Probabilistic Model-Based Planning for Human-Robot Interaction (2017, methodology): “We use long short-term memory (LSTM) RNN cells [33], and in addition to typical model training techniques (e.g., recurrent dropout regularization [34] and hyperparameter annealing), we employ the method introduced in [35] to prevent prediction errors from severely cascading into the future.”
A Transdisciplinary Review of Deep Learning Research and Its Relevance for Water Resources Scientists (2017, background): “Fang et al. (2017; hereafter termed FSKY17) applied the dropout of both Gal and Ghahramani (2016) and Semeniuta et al. (2016).”