Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations

By David Krueger, Tegan Maharaj, J'anos Kram'ar, Mohammad Pezeshki, Nicolas Ballas, Nan Rosemary Ke, Anirudh Goyal, Yoshua Bengio, Hugo Larochelle, Aaron C. Courville and 1 more. 2016

331 citations recorded by the source corpus.

We propose zoneout, a novel method for regularizing RNNs. At each timestep, zoneout stochastically forces some hidden units to maintain their previous values. Like dropout, zoneout uses random noise to train a pseudo-ensemble, improving generalization. But by preserving instead of dropping hidden units, gradient information and state information are more readily propagated through time, as in feedforward stochastic depth networks. We perform an empirical investigation of various RNN regularizers, and find that zoneout gives significant performance improvements across tasks. We achieve competitive results with relatively simple models in character- and word-level language modelling on the Penn Treebank and Text8 datasets, and combining with recurrent batch normalization yields state-of-the-art results on permuted sequential MNIST.

Subfields: Artificial Intelligence, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

Citations in the corpus

Gaussian Error Linear Units (GELUs) (2016, background): “Also, a new RNN regularizer called zoneout stochastically multiplies inputs by one (Krueger et al., 2016).”
Pointer Sentinel Mixture Models (2016, methodology): “The variants of dropout used were zone-out (Krueger et al., 2016) and variational inference based dropout (Gal, 2015).”
One-Shot Imitation Learning (2017, methodology): “Hence, we randomly discard a subset of time steps during training, an operation we call temporal dropout, analogous to (Srivastava et al., 2014; Krueger et al., 2016).”
Unsupervised Speech Representation Learning Using WaveNet Autoencoders (2019, neutral): “We therefore introduce a dropout-inspired [50] time-jitter regularizer, also reminiscent of Zoneout [51] regularization for recurrent networks.”
Regularizing and Optimizing LSTM Language Models (2017, background): “This is reminiscent of zone-out (Krueger et al., 2016) where updates to the hidden state may fail to occur for randomly selected neurons.”
Hierarchical Multiscale Recurrent Neural Networks (2016, methodology): “The COPY operation used in our model can be related to Zoneout (Krueger et al., 2016) which is a recurrent generalization of stochastic depth (Huang et al., 2016).”
Recurrent Batch Normalization (2016, methodology): “1.32 Zoneout (Krueger et al., 2016) 1.27 HM-LSTM (Chung et al., 2016) 1.24 HyperNetworks (Ha et al., 2016) 1.22 1 shows that our batch-normalized LSTM achieves state of the art on both MNIST and p MNIST.”
Natural TTS Synthesis by Conditioning Wavenet on MEL Spectrogram Predictions (2017, methodology): “5, and LSTM layers are regularized using zoneout [26] with probability 0.”
Recurrent Independent Mechanisms (2019, methodology): “For further evidence that RIMs can achieve this out-of-distribuution, we consider the task of classifying MNIST digits as sequences of pixels (Krueger et al., 2016) and assay generalization to images of resolutions different from those seen during training.”
High-Dimensional Multivariate Forecasting with Low-Rank Gaussian Copula Processes (2019, methodology): “Finally, RNNs are combined with Zone-out regularization [9] and residual connections and MXNet is used as the neural network framework [15].”
An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling (2018, background): “Language modeling remains one of the primary applications of recurrent networks and many recent works have focused on optimizing LSTMs for this task (Krueger et al., 2017; Merity et al., 2017).”
Dilated Recurrent Neural Networks (2017, methodology): “In particular, dilated Vanilla outperforms the previous RNN-based state-of-the-art Zoneout [13] with a comparable number of parameters.”
DropBlock: A regularization method for convolutional networks (2018, background): “Currently, Variational Dropout [21] and ZoneOut [22] are two of the most commonly used methods to inject noise to recurrent connections.”
Semi-supervised Training for Improving Data Efficiency in End-to-end Speech Synthesis (2018, methodology): “We use a baseline Tacotron architecture specified in [8], where we use a GMM attention [9], LSTM-based decoder with zoneout regularization [10] and phoneme inputs derived from normalized text.”
Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis (2018, methodology): “We use phoneme inputs to speed up training, and slightly change the decoder, replacing GRU cells with two layers of 256-cell LSTMs; these are regularized using zoneout (Krueger et al., 2017) with probability 0.”
Quasi-Recurrent Neural Networks (2016, methodology): “The need for an effective regularization method for LSTMs, and dropout’s relative lack of efﬁcacy when applied to recurrent connections, led to the development of recurrent dropout schemes, including variational inference–based dropout (Gal & Ghahramani, 2016) and zoneout (Krueger et al., 2016).”