On the inductive bias of dropout

By David P. Helmbold, Philip M. Long. 2014

74 citations recorded by the source corpus.

Dropout is a simple but effective technique for learning in neural networks and other settings. A sound theoretical understanding of dropout is needed to determine when dropout should be applied and how to use it most effectively. In this paper we continue the exploration of dropout as a regularizer pioneered by Wager, et.al. We focus on linear classification where a convex proxy to the misclassification loss (i.e. the logistic loss used in logistic regression) is minimized. We show: (a) when the dropout-regularized criterion has a unique minimizer, (b) when the dropout-regularization penalty goes to infinity with the weights, and when it remains bounded, (c) that the dropout regularization can be non-monotonic as individual weights increase from 0, and (d) that the dropout regularization penalty may not be convex. This last point is particularly surprising because the combination of dropout regularization with any convex loss proxy is always a convex function. In order to contrast dropout regularization with $L_2$ regularization, we formalize the notion of when different sources are more compatible with different regularizers. We then exhibit distributions that are provably more compatible with dropout regularization than $L_2$ regularization, and vice versa. These sources provide additional insight into how the inductive biases of dropout and $L_2$ regularization differ. We provide some similar results for $L_1$ regularization.

Subfields: Artificial Intelligence, Computational Mechanics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

Citations in the corpus

Formal Guarantees on the Robustness of a Classifier against Adversarial Manipulation (2017, background): “Dropout can also be interpreted as a form of regularization of the weights [22, 10].”
The Implicit and Explicit Regularization Effects of Dropout (2020, background): “The esoteric nature of dropout has inspired a large body of work studying its regularization eﬀects: Wager et al. (2013); Helmbold & Long (2015); Cavazza et al. (2017); Mianjy et al. (2018); Mianjy & Arora (2019) study dropout for linear models, matrix factorization, and linearized networks; Arora…”
How Does Mixup Help With Robustness and Generalization? (2020, methodology): “Similar approximations have been proposed before to study the regularization effect of Dropout training, see Wang & Manning (2013); Wager et al. (2013); Mianjy et al. (2018); Wei et al. (2020).”
Deep learning for short-term traffic flow prediction (2016, methodology): “From our experiments with the traffic data, we found that using sparse linear model estimation to identify spatial-temporal relations between variables yields better results, as compared to using dropout or regularization terms for neural network loss function Pachitariu and Sahani [2013], Helmbold ”
Surprising properties of dropout in deep networks (2016, methodology): “This abstracts away sampling and optimization issues to focus on the inductive bias, as in some previous work (Breiman, 2004; Zhang, 2004; Bartlett et al., 2006; Long and Servedio, 2010; Helmbold and Long, 2015).”
Data Augmentation via Levy Processes (2016, methodology): “…of dropout: using PAC-Bayes framework (McAllester, 2013), comparing dropout to “altitude training” (Wager et al., 2014), and interpreting dropout as a form of adaptive regularization (Baldi and Sadowski, 2014; Bishop, 1995; Helmbold and Long, 2015; Josse and Wager, 2014; Wager et al., 2013).”