Dropout Training as Adaptive Regularization

By Stefan Wager, Sida I. Wang, Percy Liang. 2013

645 citations recorded by the source corpus.

Dropout and other feature noising schemes control overfitting by artificially corrupting the training data. For generalized linear models, dropout performs a form of adaptive regularization. Using this viewpoint, we show that the dropout regularizer is first-order equivalent to an L2 regularizer applied after scaling the features by an estimate of the inverse diagonal Fisher information matrix. We also establish a connection to AdaGrad, an online learning algorithm, and find that a close relative of AdaGrad operates by repeatedly solving linear dropout-regularized problems. By casting dropout as regularization, we develop a natural semi-supervised algorithm that uses unlabeled data to create a better adaptive regularizer. We apply this idea to document classification tasks, and show that it consistently boosts the performance of dropout training, improving on state-of-the-art results on the IMDB reviews dataset.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

On The Impact of Machine Learning Randomness on Group Fairness (2023, background): “For example, data augmentation to increase dataset size [31], dropout to perform regularization [58], gradient noise for private training [14], etc.”
Discriminative Unsupervised Feature Learning with Convolutional Neural Networks (2014, neutral): “We ﬁrst present a formal analysis of the objective, separating it into a well deﬁned classiﬁcation problem and a regularizer that enforces invariance (resembling the analysis in Wager et al. [20]).”
Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning (2017, background): “Several studies have also conﬁrmed that this philosophy of making the predictor robust against random and local perturbation is also effective in semi-supervised learning [3], [21], [31], [41].”
Learning with Pseudo-Ensembles (2014, methodology): “The authors of [21] show that, assuming a noise process ξ such that Eξ[f(x; ξ)] = f(x), logistic regression under the influence of dropout optimizes the following objective:”
Data Recombination for Neural Semantic Parsing (2016, background): “Dropout training has been shown to be a form of adaptive regularization (Hinton et al., 2012; Wager et al., 2013).”
Collaborative Deep Learning for Recommender Systems (2014, methodology): “For CDL with more than one layer of SDAE (L > 2), we use a dropout rate [2, 33, 11] of 0.”
Missing Data Imputation for Supervised Learning (2016, background): “4 } according to the MCAR mechanism While perturbation is used primarily to study the eﬀect of larger amounts of missing data, we note that missing-data perturbation is a form of dropout noise that can be used to control overﬁtting during the training process (Wager et al., 2013).”
Deep Learning: Methods and Applications (2014, methodology): “The first is a powerful regularization technique called “dropout”; see details in [166] and a series of further analysis and improvement in [10, 13, 240, 381, 385].”
Conditional Computation in Neural Networks for faster models (2015, background): “Dropout can be interpreted as a form of regularization to prevent overfitting (Wager et al., 2013).”
A Primer on Neural Network Models for Natural Language Processing (2015, methodology): “At each input state, a gate is used to decide how much of the new input should be written to the memory cell, and how much of the current content of the memory cell should be forgotten.”
Adversarial Dropout for Supervised and Semi-supervised Learning (2017, methodology): “Corrupting training data with noises has been well-known to be a method to stabilize prediction (Bishop 1995a; Maaten et al. 2013; Wager, Wang, and Liang 2013).”
Enhancing Performance of Deep Learning Models with different Data Augmentation Techniques: A Survey (2020, background): “By doing this, the network will become more robust by learning feature [29].”
Data augmentation instead of explicit regularization (2018, methodology): “Dropout has been extensively used and studied as a regularization method for neural networks (Wager et al., 2013), but the exact way in which dropout may improve generalization is still an open question and it has been concluded that the effects of dropout on neural networks are somewhat mysterious,”
Follow the Leader with Dropout Perturbations (2014, methodology): “Standard Dropout Perturbations Dropout is normally defined as dropping hidden units or features in a neural network while training the parameters of the network on batch data using Gradient Descent (Hinton et al., 2012; Wang and Manning, 2013; Wager et al., 2013).”
Using Wasserstein Generative Adversarial Networks for the design of Monte Carlo simulations (2019, positive): “Additionally, the resulting data generating distributions can be shown to satisfy certain privacy guarantees with respect to the data they were trained on under some modifications, see Xie et al. [2018]. This would allow the scientific community to benefit from otherwise inaccessible confidential da”
Surprising properties of dropout in deep networks (2016, methodology): “With this compensation, the dropout can be viewed as injecting zero-mean additive noise at each non-output node (Wager et al., 2013).”
Does Data Augmentation Lead to Positive Margin? (2019, methodology): “Wager et al. (2013) show that training generalized linear models while randomly dropping features is approximately equivalent to (cid:96) 2 -regularization normalized by the inverse diagonal Fisher information matrix.”