Understanding Dropout

By Pierre Baldi, Peter Sadowski. 2013

785 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

Collaborative Deep Learning for Recommender Systems (2014, methodology): “For CDL with more than one layer of SDAE (L > 2), we use a dropout rate [2, 33, 11] of 0.”
Deep Learning: Methods and Applications (2014, methodology): “The first is a powerful regularization technique called “dropout”; see details in [166] and a series of further analysis and improvement in [10, 13, 240, 381, 385].”
Overcoming Catastrophic Forgetting by Incremental Moment Matching (2017, background): “There are studies [27, 20] that have interpreted dropout as an exponential ensemble of weak learners.”
Understanding the Role of Training Regimes in Continual Learning (2020, background): “Dropout [27] is a well-established technique in deep learning, which is also well-studied theoretically [3, 70, 26, 18, 74].”
Dropout as an Implicit Gating Mechanism For Continual Learning (2020, background): “[3] showed the neural activity of several biological brains, which is in line with the neural activity in dropout networks, as shown by [2] and our experiments.”
Adversarial Dropout for Supervised and Semi-supervised Learning (2017, methodology): “Extensive empirical research proved the effectiveness of dropout techniques (Krizhevsky, Sutskever, and Hinton 2012; Ma and Hovy 2016), and dropout has recently been analyzed theoretically to reveal the rational of how and why dropout works (Srivastava et al. 2014; Baldi and Sadowski 2013; Wager, Wa”
Avoiding pathologies in very deep networks (2014, background): “Baldi and Sadowski (2013) and Wang and Manning (2013) analyzed dropout in terms of the effective prior induced by this procedure in several models, such as linear and logistic regression.”
Dropout-GAN: Learning from a Dynamic Ensemble of Discriminators (2018, positive): “In a disparate line of work, dropout was introduced by [18] and it has been proven to be a very useful and widely used technique in neural networks to prevent overfitting [5, 11, 37].”
Uncertainty in Deep Learning (2016, background): “Dropout in linear networks was also studied in [Baldi and Sadowski, 2013].”
Robustly representing inferential uncertainty in deep neural networks through sampling (2016, background): “5 , which corresponds to the best value for regularizing a linear layer Baldi & Sadowski (2013).”
Deep learning for visual understanding: A review (2016, neutral): “Dropout was proposed by Hin-ton et al. [38] and explained in-depth by Baldi et al. [39].”
Learning with Pseudo-Ensembles (2014, methodology): “e of V i. Note that for Eq. 2 to act on the “variance” of the fi (x;˘), we should have f i (x) ˇE ˘f (x;˘). This approximation holds reasonably well for many useful neural network architectures [1, 22]. In our experiments we actually compute the penalties V ibetween independently-sampled pairs of”
Ensemble deep learning in bioinformatics (2020, background): “For example, the random deactivation of neurons, termed dropout, originally proposed as a regularization strategy 30 for addressing model overfitting is now widely known as an implicit ensemble strategy 31,32 .”
The dropout learning algorithm (2014, methodology): “Finally, since the posting of [27], three articles with dropout in their title were presented at the NIPS 2013 conference: a training method based on overlaying a dropout binary belief network on top of a neural network [7]; an analysis of the adaptive regularizing properties of dropout in the shall”
Pushing the bounds of dropout (2018, methodology): “We offer no solution to this bias issue, but refer the reader to previous studies of dropout’s approximation properties such as (Baldi & Sadowski 2013) and our subsequent empirical results.”
Dropout distillation (2016, background): “Some works supported the view of dropout performing stochastic gradient descent of some regularized loss function (Baldi & Sadowski, 2013), or as an adaptive L2 regularizer (Wager et al., 2013), or more recently as an approximation to a well-known Bayesian model, namely the deep Gaussian process…”
A Bayesian encourages dropout (2014, background): “The success of dropout attracts many researchers’ attention and several theoretical studies have been performed aiming for better understanding of dropout (Wager et al., 2013; Baldi & Sadowski, 2013; Ba & Frey, 2013).”
Deep learning in bioinformatics (2016, background): “Dropout randomly removes hidden units from neural networks during training and can be considered an ensemble of possible subnetworks [52].”
Deep Convolutional Neural Networks for Image Classification: A Comprehensive Review (2017, methodology): “…characteristics; its efficiency and ensemble learning characteristics were examined by Warde-Farley, Goodfellow, Courville, and Bengio (2013), while Baldi and Sadowski (2013, 2014) provided a detailed mathematical analysis of its static and dynamic properties and characterized its averaging…”