Fast dropout training

By Sida I. Wang, Christopher D. Manning. 2013

473 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

Stochastic Training of Graph Convolutional Networks (2017, methodology): “We analyze their variance in a simple independent Gaussian case, where we assume that activations are Gaussian random variables 2We will omit the subscripts CV and i in subsequent sections when there is no confusion. hv ∼ N (µv, σ2v) following Wang & Manning (2013).”
Convolutional Neural Networks for Sentence Classification (2014, background): “G-Dropout, F-Dropout : Gaussian Dropout and Fast Dropout from Wang and Manning (2013).”
Recurrent Neural Network Regularization (2014, background): “While much work has extended dropout in various ways (Wang & Manning, 2013; Wan et al., 2013), there has been relatively little research in applying it to RNNs.”
Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations (2016, background): “We conjecture that identity-masking is more appropriate for RNNs, since it makes it easier for the network to preserve information from previous time-steps going forwards, and it facilitates, rather than hinders, the ﬂow of gradient information going backwards.”
Dropout Improves Recurrent Neural Networks for Handwriting Recognition (2013, methodology): “Due to the impressive performance of dropout, some extensions of this technique were proposed, including DropConnect [18], Maxout networks [19], and an approximate approach for fast training with dropout [20].”
Dropout Training as Adaptive Regularization (2013, methodology): “Note that”
A Comprehensive guide to Bayesian Convolutional Neural Network with Variational Inference (2019, background): “Several authors have claimed that Dropout [58] and Gaussian Dropout [61] can be viewed as approximate variational inference schemes [13, 32].”
To Drop or Not to Drop: Robustness, Consistency and Differential Privacy Properties of Dropout (2015, background): “…neural networks, there have been a series of works providing strong theoretical understanding of the dropout heuristic Baldi and Sadowski [2014], Wager et al. [2013], Wang et al. [2013], van Erven et al. [2014], Helmbold and Long [2014], Wager et al. [2014], McAllester [2013], Maaten et al. [2013].”
Deep learning in neural networks: An overview (2014, background): “A deterministic approximation coined fast dropout (Wang & Manning, 2013) can lead to faster learning and evaluation and was adapted for RNNs (Bayer, Osendorfer, Chen, Urban, & van der Smagt, 2013).”
Missing Data Imputation for Supervised Learning (2016, background): “Future work might compare missing-data perturbation with dropout training, which changes to zero all the values of a random subset of features (Hinton et al., 2012; Maaten et al., 2013; Wang and Manning, 2013).”
Recent advances in convolutional neural networks (2015, methodology): “following subsection, we introduce two effective regularization techniques: Dropout [16], [49], [50] and DropConnect [28].”
Multiplicative Normalizing Flows for Variational Bayesian Neural Networks (2017, methodology): “By employing the Central Limit Theorem 7 we can express the distribution of the activations as a Gaussian (Wang & Manning, 2013) with variance affected by the variance of the Bernoulli r.v., V ( z ) = π (1 − π ) .”
Variational Dropout and the Local Reparameterization Trick (2015, methodology): “Making use of this fact, [22] proposed Gaussian dropout, a regularization method where, instead of applying (9), the activations are directly drawn from their (approximate or exact) marginal distributions as given by (10).”
Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Models (2019, background): “We provide a theoretical understanding of dropout and its variants, such as Gaussian dropout (Wang & Manning, 2013), variational dropout (Kingma et al., 2015), and drop-connect (Wan et al., 2013), as an adaptive L 2 -penalty toward the origin (all zero parameters 0 ) and generalize dropout by…”