A Bayesian encourages dropout

By Shin-ichi Maeda. 2014

45 citations recorded by the source corpus.

Dropout is one of the key techniques to prevent the learning from overfitting. It is explained that dropout works as a kind of modified L2 regularization. Here, we shed light on the dropout from Bayesian standpoint. Bayesian interpretation enables us to optimize the dropout rate, which is beneficial for learning of weight parameters and prediction after learning. The experiment result also encourages the optimization of the dropout.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Variational Dropout Sparsifies Deep Neural Networks (2017, background): “Other Bayesian interpretations of dropout training have also appeared during past several years (Maeda, 2014; Gal & Ghahramani, 2015; Srinivas & Babu, 2016).”
Variational Dropout and the Local Reparameterization Trick (2015, background): “Recently, [15] proposed another Bayesian perspective on dropout.”
The Implicit and Explicit Regularization Effects of Dropout (2020, background): “Wang & Manning (2013); Maeda (2014); Gal & Ghahramani (2016a); Ma et al. (2016) study dropout from a Bayesian perspective.”
Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles (2016, background): “There has been work on approximate Bayesian interpretation (Gal and Ghahramani, 2016; Kingma et al., 2015; Maeda, 2014) of dropout.”
Uncertainty in Deep Learning (2016, methodology): “5 In the dropout case, initial research studying the technique from a Bayesian perspective includes [Ba and Frey, 2013; Maeda, 2014; Wang and Manning, 2013].”
Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning (2015, background): “This extends on initial research studying dropout from the Bayesian perspective (Wang & Manning, 2013; Maeda, 2014).”
Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning (2017, methodology): “From a Bayesian perspective, dropout is a method to introduce prior distribution for the parameters rather than a regularization that introduces perturbation to the inputs and the hidden variables [6], [26].”
Dropout Feature Ranking for Deep Learning Models (2017, background): “Variational dropout [19] optimizes the dropout rate θ as a parameter instead of it being a ﬁxed hyperpa-rameter.”
Regularization for Deep Learning: A Taxonomy (2017, background): “…but also several variants of Dropout have been proposed that provide additional theoretical motivation and improved empirical results (Standout (Ba and Frey, 2013), Random dropout probability (Bouthillier et al., 2015), Bayesian dropout (Maeda, 2014), Test-time dropout (Gal and Ghahramani, 2016)).”