Generalized Dropout

By Suraj Srinivas, R. Venkatesh Babu. 2016

49 citations recorded by the source corpus.

Deep Neural Networks often require good regularizers to generalize well. Dropout is one such regularizer that is widely used among Deep Learning practitioners. Recent work has shown that Dropout can also be viewed as performing Approximate Bayesian Inference over the network parameters. In this work, we generalize this notion and introduce a rich family of regularizers which we call Generalized Dropout. One set of methods in this family, called Dropout++, is a version of Dropout with trainable parameters. Classical Dropout emerges as a special case of this method. Another member of this family selects the width of neural network layers. Experiments show that these methods help in improving generalization performance over Dropout.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Variational Dropout Sparsifies Deep Neural Networks (2017, background): “Generalized Dropout (Srinivas & Babu, 2016) provides a way to tune individual dropout rates for neurons, but uses a biased gradient estimator.”
Bayesian Compression for Deep Learning (2017, background): “Table 1: Learned architectures with Sparse VD [51], Generalized Dropout (GD) [66] and Group Lasso (GL) [73].”
Compressing Neural Networks using the Variational Information Bottleneck (2018, methodology): “…benchmarks, we compare our VIBNet with published results from a variety of contemporary state-of-the-art methods including Generalized Dropout (GD) (Srinivas & Babu, 2016), Group Lasso (GL) (Wen et al., 2016), Sparse Variational Dropout (VD) (Molchanov et al., 2017), Bayesian Compression with…”
Channel Selection Using Gumbel Softmax (2020, background): “[43] uses a form of trainable dropout, learning a per-neuron gating probability.”
The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks (2018, background): “Bayesian perspectives on dropout learn dropout probabilities during training (Gal et al., 2017; Kingma et al., 2015; Srinivas & Babu, 2016).”
An end-to-end approach for speeding up neural network inference (2018, background): “[44] uses a form of trainable dropout, learning a per-neuron gating probability.”
Picking Winning Tickets Before Training by Preserving Gradient Flow (2020, background): “Srinivas & Babu (2016) Dettmers & Zettlemoyer (2019), follow a prune-redistribute-regrowth cycle for pruning.”
Sanity-Checking Pruning Methods: Random Tickets can Win the Jackpot (2020, background): “Several other criterions and settings are proposed for unstructured pruning[5, 28, 36].”