The Implicit and Explicit Regularization Effects of Dropout

By Colin Wei, Sham M. Kakade, Tengyu Ma. 2020

130 citations recorded by the source corpus.

Dropout is a widely-used regularization technique, often required to obtain state-of-the-art for a number of architectures. This work demonstrates that dropout introduces two distinct but entangled regularization effects: an explicit effect (also studied in prior work) which occurs since dropout modifies the expected training objective, and, perhaps surprisingly, an additional implicit effect from the stochasticity in the dropout training update. This implicit regularization effect is analogous to the effect of stochasticity in small mini-batch stochastic gradient descent. We disentangle these two effects through controlled experiments. We then derive analytic simplifications which characterize each effect in terms of the derivatives of the model and the loss, for deep neural networks. We demonstrate these simplified, analytic regularizers accurately capture the important aspects of dropout, showing they faithfully replace dropout in practice.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Understanding the Role of Training Regimes in Continual Learning (2020, background): “[74] showed that dropout has both implicit and explicit but entangled regularization effects: More specifically, they provided an approximation for the explicit effect on the i-th hidden layer (denoted by hi) under input x by: (p/p−1)[∇ 2 hi L] [diag(h(2)i )], where p is the dropout probability, L i”
Why Do Better Loss Functions Lead to Less Transferable Features? (2020, background): “Dropout produces both implicit regularization, by introducing noise into the optimization process [75], and explicit regularization, by changing the parameters that minimize the loss [69].”
Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training (2023, unknown): “To the best of our knowledge, this estimator of Gauss-Newton matrix was ﬁrst used in (Wei et al., 2020).”
Regularizing Neural Networks via Adversarial Model Perturbation (2020, background): “Some recent research in fact suggests that the effectiveness of certain regularization techniques is due to their ability to find flatter minima [22, 44].”
JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models (2024, background): “In releasing the jailbreak artifacts, we hope to expedite future research on jailbreaking, especially on the defense side.”
Implicit Regularization in Deep Learning May Not Be Explainable by Norms (2020, background): “[69] Colin Wei, Sham Kakade, and Tengyu Ma.”
Drawing Multiple Augmentation Samples Per Image During Training Efficiently Decreases Test Error (2021, methodology): “Our study was inspired by a recent study of the origin of the regularization beneﬁts of dropout [Wei et al., 2020].”
R-Drop: Regularized Dropout for Neural Networks (2021, methodology): “Compared with the loss without dropout L = 1 n ∑n i=1− logP(yi|xi), it has been shown that optimizing the loss LNLL constrains the model complexity by controlling the Jacobian matrix of the model P ξ (·) [66].”
How Does Mixup Help With Robustness and Generalization? (2020, methodology): “This technique is then extended to drive more properties of Dropout, including the inductive bias of Dropout (Helmbold & Long, 2015), the regularization effect in matrix factorization (Mianjy et al., 2018), and the implicit regularization in neural networks (Wei et al., 2020).”