An empirical analysis of dropout in piecewise linear networks

By David Warde-Farley, Ian J. Goodfellow, Aaron C. Courville, Yoshua Bengio. 2013

111 citations recorded by the source corpus.

The recently introduced dropout training criterion for neural networks has been the subject of much attention due to its simplicity and remarkable effectiveness as a regularizer, as well as its interpretation as a training procedure for an exponentially large ensemble of networks that share parameters. In this work we empirically investigate several questions related to the efficacy of dropout, specifically as it concerns networks employing the popular rectified linear activation function. We investigate the quality of the test time weight-scaling inference procedure by evaluating the geometric average exactly in small models, as well as compare the performance of the geometric mean to the arithmetic mean more commonly employed by ensemble techniques. We explore the effect of tied weights on the ensemble interpretation by training ensembles of masked networks without tied weights. Finally, we investigate an alternative criterion based on a biased estimator of the maximum likelihood ensemble gradient.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Deep Learning: Methods and Applications (2014, methodology): “[385] analyzed the disentangling effects of dropout and showed that it helps because different members of the bag share parameters.”
BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning (2020, background): “, 2014) can be interpreted as creating an exponential number of weight-sharing sub-networks, which are implicitly ensembled in test time prediction (Warde-Farley et al., 2014).”
An Overview of Overfitting and its Solutions (2019, background): “In the work of David Warde-Farley, Ian J. Goodfellow, Aaron Courville, and Yoshua Bengio [20], they investigated the efﬁcacy of dropout, focusing on the speciﬁc case of the popular rectiﬁed linear nonlinearity for hidden units.”
Dropout-GAN: Learning from a Dynamic Ensemble of Discriminators (2018, unknown): “In a disparate line of work, dropout was introduced by [18] and it has been proven to be a very useful and widely used technique in neural networks to prevent overfitting [5, 11, 37].”
Learning with Pseudo-Ensembles (2014, background): “Also, by acting strictly to make the output of the parent model more robust to ξ-perturbation, the performance of this regularizer rebuts the claim in [22] that noise-robustness plays only a minor role in the success of standard dropout.”
Deep learning for visual understanding: A review (2016, methodology): “This method was further improved in [40 – 45].”
Self-Attentive Sequential Recommendation (2018, background): “Further analysis points out that dropout can be viewed as a form of ensemble learning which considers an enormous number of models (exponential in the number of neurons and input features) that share parameters [39].”
Rectified Factor Networks (2015, methodology): “, 2013) and dropout (Hinton et al., 2012; Warde-Farley et al., 2013; Baldi & Sadowski, 2014) are techniques to obtain very sparse representations.”
Towards dropout training for convolutional neural networks (2015, background): “…for linear and sigmoidal networks (Baldi & Sadowski, 2014; Wager el al., 2013); for piecewise linear networks such as rectified linear networks, Warde et al. (2014) empirically showed that weight-scaling approximation is a remarkable and accurate surrogate for the true geometric mean, by…”
Deep learning for visual understanding (2017, unknown): “[53] analyzed the e cacy of dropouts and suggested that dropout is an extremely e ective ensemble learning method.”
Pushing the bounds of dropout (2018, background): “The dominant perspective today views dropout as either an implicit ensemble method (Warde-Farley et al. 2013) or averaging over an approximate Bayesian posterior (Gal & Ghahramani 2016a).”
The Rhetoric and Reality of Anthropomorphism in Artificial Intelligence (2019, methodology): “Although bagging is not typically used with neural networks, the method is often compared to a popular regularization technique for DNNs known as dropout (Hin-ton et al. 2012; Srivastava et al. 2014; Warde-Farley et al. 2013).”