On Fast Dropout and its Applicability to Recurrent Networks

By Justin Bayer, Christian Osendorfer, Nutan Chen, Sebastian Urban, Patrick van der Smagt. 2013

66 citations recorded by the source corpus.

Recurrent Neural Networks (RNNs) are rich models for the processing of sequential data. Recent work on advancing the state of the art has been focused on the optimization or modelling of RNNs, mostly motivated by adressing the problems of the vanishing and exploding gradients. The control of overfitting has seen considerably less attention. This paper contributes to that by analyzing fast dropout, a recent regularization method for generalized linear models and neural networks from a back-propagation inspired perspective. We show that fast dropout implements a quadratic form of an adaptive, per-parameter regularizer, which rewards large weights in the light of underfitting, penalizes them for overconfident predictions and vanishes at minima of an unregularized training loss. The derivatives of that regularizer are exclusively based on the training error signal. One consequence of this is the absense of a global weight attractor, which is particularly appealing for RNNs, since the dynamics are not biased towards a certain regime. We positively test the hypothesis that this improves the performance of RNNs on four musical data sets.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

Citations in the corpus

Deep learning in neural networks: An overview (2014, background): “A deterministic approximation coined “fast dropout” (Wang and Manning, 2013) can lead to faster learning and evaluation and was adapted for RNNs (Bayer et al., 2013).”
A Theoretically Grounded Application of Dropout in Recurrent Neural Networks (2015, background): “[8] apply a deterministic approximation of dropout (fast dropout) in RNNs.”
Uncertainty in Deep Learning (2016, methodology): “This is an interesting use of the ideas above; existing literature in RNNs has established that dropout cannot be applied in RNNs apart from the forward connections [Bayer et al., 2013; Bluche et al., 2015; Pachitariu and Sahani, 2013; Pham et al., 2014; Zaremba et al., 2014].”
Regularizing Recurrent Networks - On Injected Noise and Norm-based Methods (2014, background): “The resulting models had orders of magnitude better training times than a naive dropout approach, and the test results matched, and were sometimes better than those of plain MLPs. Bayer et al. (2013) veriﬁed the validity of the fast-dropout approach on RNNs.”
Pushing the bounds of dropout (2018, methodology): “Bayer et al. (2013) echo this claim about RNNs, which is then cited by Zaremba et al. (2014), but their work is based on LSTMs not standard RNNs. Finally, Gal & Ghahramani (2016b) cite all of the above but also work with LSTMs.”
Imposing higher-level Structure in Polyphonic Music Generation using Convolutional Restricted Boltzmann Machines and Constraints (2016, background): “State-of-the-art connectionist models for sequential data, although not explicitly dedicated to music modeling, are increasingly often tested in the musical domain [Bayer et al. 2014; Bengio et al. 2013; Pascanu et al. 2013].”
How to Construct Deep Recurrent Neural Networks (2013, methodology): “On every music dataset the performance by this model is significantly better than those achieved by all the other models as well as the best results reported with recurrent neural networks in (Bayer et al., 2013).”
Variational inference of latent state sequences using Recurrent Networks (2014, methodology): “The same split into training, validation and test sets as done in the previous publications relying on that benchmark [20, 3, 1] was used.”
Curriculum Dropout (2017, methodology): “Leveraging on the Dropout idea, many works have proposed variations of the original strategy [14, 22, 31, 30, 1, 20].”
Recurrent Neural Network Regularization (2014, background): “Bayer et al. (2013) claim that conventional dropout does not work well with RNNs because the recurrence ampliﬁes noise, which in turn hurts learning.”
DeepCare: A Deep Dynamic Memory Model for Predictive Medicine (2016, background): “However, the original version of dropout does not work well with RNNs because it may hurt the dependencies in sequential data during training [54, 55].”
Scalable Bayesian Learning of Recurrent Neural Networks for Language Modeling (2016, background): “Future works include improving the testing efficiency for the large-scale RNNs, via learning a single neural network that approximates the model averaging result (Korattikara et al., 2015).”
Understanding Generalization in Recurrent Neural Networks (2020, methodology): “For example, Bayer et al. (2013) claimed that conventional dropout did not work well with RNNs because the recurrence ampliﬁed noise, which in turn hurt learning.”