To Drop or Not to Drop: Robustness, Consistency and Differential Privacy Properties of Dropout

By Prateek Jain, Vivek Kulkarni, Abhradeep Thakurta, Oliver Williams. 2015

33 citations recorded by the source corpus.

Training deep belief networks (DBNs) requires optimizing a non-convex function with an extremely large number of parameters. Naturally, existing gradient descent (GD) based methods are prone to arbitrarily poor local minima. In this paper, we rigorously show that such local minima can be avoided (upto an approximation error) by using the dropout technique, a widely used heuristic in this domain. In particular, we show that by randomly dropping a few nodes of a one-hidden layer neural network, the training objective function, up to a certain approximation error, decreases by a multiplicative factor. On the flip side, we show that for training convex empirical risk minimizers (ERM), dropout in fact acts as a "stabilizer" or regularizer. That is, a simple dropout based GD method for convex ERMs is stable in the face of arbitrary changes to any one of the training points. Using the above assertion, we show that dropout provides fast rates for generalization error in learning (convex) generalized linear models (GLM). Moreover, using the above mentioned stability properties of dropout, we design dropout based differentially private algorithms for solving ERMs. The learned GLM thus, preserves privacy of each of the individual training points while providing accurate predictions for new test points. Finally, we empirically validate our stability assertions for dropout in the context of convex ERMs and show that surprisingly, dropout significantly outperforms (in terms of prediction accuracy) the L2 regularization based methods for several benchmark datasets.

Subfields: Artificial Intelligence, Computer Science Applications, Sociology and Political Science, Optimization & Training, Safety, Robustness & Interpretability, Privacy & Security

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

In Sociology and Political Science

In Optimization & Training

In Safety, Robustness & Interpretability

In Privacy & Security

Citations in the corpus

Deep Models Under the GAN: Information Leakage from Collaborative Deep Learning (2017, background): “Several works have adopted DP as an efficient defense mechanism such as [5, 7, 11, 12, 18, 23, 24, 37, 53, 60, 65, 72, 86].”
Adversarial Dropout for Supervised and Semi-supervised Learning (2017, background): “Extending the weight sharing perspective, several studies (Baldi and Sadowski 2013; Chen et al. 2014; Jain et al. 2015) analyzed the ensemble effects from the dropout.”
Demystifying Dropout (2019, background): “Besides, to disclose the success of dropout, various works (Hinton et al., 2012; Wager et al., 2013; Bouthillier et al., 2015; Helmbold & Long, 2017; Jain et al., 2015) have attempted to analyze it from different perspectives.”
Stolen Memories: Leveraging Model Memorization for Calibrated White-Box Membership Inference (2019, result): “Generalization alone is not sufficient to protect against membership inference [47], and in fact, our empirical results (Section 5) show that we can successfully attack even models with negligible generalization error; however, dropout has been shown not only to reduce overfitting, but to strengthen”
Dropout distillation (2016, background): “The latter property arises from the ability of dropout training to escape poor, local minima during the optimization of the empirical risk (Jain et al., 2015).”
Privacy-preserving deep learning (2015, background): “Recent results show that a noisy variant of stochastic gradient descent achieves optimal error for minimizing Lipschitz convex functions over `2-bounded sets [4], and that randomized “dropout,” used to prevent overfitting, cal also strengthen the privacy guarantee in a simple 1-layer neural network ”
Regularizing Deep Neural Networks by Noise: Its Interpretation and Optimization (2017, background): “On the other hand, [36] views dropout as an adaptive regularizer for generalized linear models and [16] claims that dropout is effective to escape local optima for training deep neural networks.”