The dropout learning algorithm

By Pierre Baldi, Peter Sadowski. 2014

343 citations recorded by the source corpus.

Dropout is a recently introduced algorithm for training neural network by randomly dropping units during training to prevent their co-adaptation. A mathematical analysis of some of the static and dynamic properties of dropout is provided using Bernoulli gating variables, general enough to accommodate dropout on units or connections, and with variable rates. The framework allows a complete analysis of the ensemble averaging properties of dropout in linear networks, which is useful to understand the non-linear case. The ensemble averaging properties of dropout in non-linear logistic networks result from three fundamental equations: (1) the approximation of the expectations of logistic functions by normalized geometric means, for which bounds and estimates are derived; (2) the algebraic equality between normalized geometric means of logistic functions with the logistic of the means, which mathematically characterizes logistic functions; and (3) the linearity of the means with respect to sums, as well as products of independent variables. The results are also extended to other classes of transfer functions, including rectified linear functions. Approximation errors tend to cancel each other and do not accumulate. Dropout can also be connected to stochastic neurons and used to predict firing rates, and to backpropagation by viewing the backward propagation as ensemble averaging in a dropout linear network. Moreover, the convergence properties of dropout can be understood in terms of stochastic gradient descent. Finally, for the regularization properties of dropout, the expectation of the dropout gradient is the gradient of the corresponding approximation ensemble, regularized by an adaptive weight decay term with a propensity for self-consistent variance minimization and sparse representations.

Subfields: Artificial Intelligence, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

Citations in the corpus

To Drop or Not to Drop: Robustness, Consistency and Differential Privacy Properties of Dropout (2015, background): “In comparison to recent works that analyze dropout for ERM style problems Baldi and Sadowski [2014], Wager et al. [2014], we study the general problem of convex ERM in generalized linear model (GLM) and provide precise generalized error bounds for the same.”
Deep learning in neural networks: An overview (2014, methodology): “Some view it as an ensemble method that trains multiple data models simultaneously (Baldi and Sadowski, 2014).”
Searching for exotic particles in high-energy physics with deep learning (2014, methodology): “neural networks with multiple hidden layers – have come from alleviating the gradient diffusion problem by a combination of factors, including: 1) speeding up the stochastic gradient descent algorithm with graphics processors; 2) using much larger training sets; 3) using new learning algorithms, inc”
Deep Sequential Neural Network (2014, background): “Many variants of learning algorithms have been proposed, from complex gradient computations [11], to dropout methods [1], but the baseline learning algorithm still consists in recursively computing the gradient by using the back-propagation algorithm and performing (stochastic) gradient descent.”
Interventional Few-Shot Learning (2020, neutral): “It is shown in [5] that Eq.”
A Transdisciplinary Review of Deep Learning Research and Its Relevance for Water Resources Scientists (2017, methodology): “Making inference using a trained deep network is akin to using the ensemble mean to make more robust decisions [Baldi and Sadowski, 2014; Hara et al., 2016]; (2) because connections are randomly set to 0, they cannot co-adapt with others during training to fit the results [Hinton et al.”
Global optimization issues in deep network regression: an overview (2018, methodology): “Among pruning methods we mention the DROPOUT algorithm for DN [9,131].”
Geometric deep learning (2016, background): “Given its popularity several follow up works have shown either improvements in terms of speed [96], applicability to recurrent networks [67], and theory [7, 47].”
On the inductive bias of dropout (2014, background): “…criterion has a unique minimizer, • when the dropout-regularization penalty goes to infinity with the weights, and when it remains bounded, • that the dropout regularization can be non-monotonic as individual weights increase from 0, and • that the dropout regularization penalty may not be convex.”
Data Augmentation via Levy Processes (2016, methodology): “…of dropout: using PAC-Bayes framework (McAllester, 2013), comparing dropout to “altitude training” (Wager et al., 2014), and interpreting dropout as a form of adaptive regularization (Baldi and Sadowski, 2014; Bishop, 1995; Helmbold and Long, 2015; Josse and Wager, 2014; Wager et al., 2013).”
Understanding Dropout (2013, background): “It can be shown that for any error function that is convex up (∪), the error of the mean, weighted geometric mean, and normalized weighted geometric mean of an ensemble is always less than the expected error of the models [1].”
Understanding Locally Competitive Networks (2014, methodology): “Note that this model of models interpretation is different from the model averaging perspective of dropout [11] which posits that using dropout is equivalent to averaging the predictions of many subnetworks on the same data.”
Causal Intervention for Weakly-Supervised Semantic Segmentation (2020, background): “, 46(3):399–424, 2011. 3 [4] Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. Segnet: A deep convolutional encoder-decoder architecture for image segmentation. TPAMI, 39(12):2481–2495, 2017.8 [5] Pierre Baldi and Peter Sadowski. The dropout learning algorithm. Artiﬁcial Intelligence, 210:78–”
Surprising properties of dropout in deep networks (2016, background): “Proof: [2] show that for networks of linear units (as opposed to the non-linear rectified linear units we focus on) the network’s output without dropout equals the expected output over dropout patterns, so in our notation: W(x) equals ER(D(W ,x,R)).”
Detecting Cardiovascular Disease from Mammograms with Deep Learning (2017, unknown): “dropout [47], [48] is applied to the input of the first FC layer.”
Visual Attention Methods in Deep Learning: An In-Depth Survey (2022, methodology): “Drop-out [8], Mirror [79], Reverse [19], Inverse [193], Reciprocal [177] weakly detection, detection and segmentation, • efficient methods Although channel attention involves similar steps as position attention, it differs because the features are used directly without passing through convolutional…”
Rectified Factor Networks (2015, methodology): “Rectiﬁed linear units (ReLU) (Nair & Hinton, 2010; Glorot et al., 2011; Zeiler et al., 2013) and dropout (Hinton et al., 2012; Warde-Farley et al., 2013; Baldi & Sadowski, 2014) are techniques to obtain very sparse representations.”
Altitude Training: Strong Bounds for Single-Layer Dropout (2014, background): “Moreover, provided that � is not too close to 1, dropout behaves similarly to an adaptive L 2 regularizer with parameter � / (1 � [6, 12], and at least in linear regression such L 2 regularization improves generalization error by a constant factor.”
Deconfounded Image Captioning: A Causal Retrospect (2020, background): “The ﬁrst step is called Normalized Weighted Geometric Mean (NWGM) approximation [30], [69], [70] which absorbs the expectations into the network (see Section 4.2).”