Concrete Dropout

By Yarin Gal, Jiri Hron, Alex Kendall. 2017

677 citations recorded by the source corpus.

Dropout is used as a practical tool to obtain uncertainty estimates in large vision models and reinforcement learning (RL) tasks. But to obtain well-calibrated uncertainty estimates, a grid-search over the dropout probabilities is necessary—a prohibitive operation with large models, and an impossible one with RL. We propose a new dropout variant which gives improved performance and better calibrated uncertainties. Relying on recent developments in Bayesian deep learning, we use a continuous relaxation of dropout’s discrete masks. Together with a principled optimisation objective, this allows for automatic tuning of the dropout probability in large models, and as a result faster experimentation cycles. In RL this allows the agent to adapt its uncertainty dynamically as more data is observed. We analyse the proposed variant extensively on a range of tasks, and give insights into common practice in the field where larger dropout probabilities are often used in deeper model layers.

Subfields: Artificial Intelligence, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Probabilistic & Bayesian Methods

Citations in the corpus

Scalable agent alignment via reward modeling: a research direction (2018, methodology): “While Bayesian methods can help with calibration (Gal et al., 2017a), they are insufﬁcient in practice for deep neural networks (Kuleshov et al., 2018).”
The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks (2018, background): “Bayesian perspectives on dropout learn dropout probabilities during training (Gal et al., 2017; Kingma et al., 2015; Srinivas & Babu, 2016).”
Dynamic Model Pruning with Feedback (2020, background): “Gal et al. (2017); Neklyudov et al. (2017); Molchanov et al. (2017) prune from Bayesian perspectives to learn dropout probabilities during training to prune and sparsify networks as dropout weight probabilities reach 1.”
A Simple Baseline for Bayesian Uncertainty in Deep Learning (2019, background): “Concrete dropout [10] extends this idea to optimize the dropout probabilities as well.”
Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models (2018, methodology): “Whilst accurate Bayesian NN inference is possible with sufficient compute [10], approximate inference methods [45], [38], [46] have enjoyed recent popularity given their simpler implementation and faster training times.”
Calibrating Deep Convolutional Gaussian Processes (2018, background): “This might be the reason why these parameters were learned through cross-validation in [10].”
Towards Safe Autonomous Driving: Capture Uncertainty in the Deep Neural Network For Lidar 3D Vehicle Detection (2018, background): “concrete dropout [31]) as an interesting future work.”
Regularisation of neural networks by enforcing Lipschitz continuity (2018, background): “Another modification, concrete dropout (Gal et al., 2017), allows one to directly learn the dropout rate, p, thus making the search for a good set of hyperparameters easier.”
Ensemble Bayesian Decision Making with Redundant Deep Perceptual Control Policies (2018, methodology): “C. Implementation To get a better calibrated uncertainty measure, we used Concrete Dropout [21], which allows for automatic tuning of the dropout probability in large models.”
Accurate Uncertainties for Deep Learning Using Calibrated Regression (2018, methodology): “Recently, Gal et al. (2017) and Lakshminarayanan et al. (2017) proposed uncertainty estimation techniques for deep neural networks, which include ensemble methods, het-eroscedastic regression, and concrete dropout.”
Randomized Prior Functions for Deep Reinforcement Learning (2018, background): “2Concrete dropout assymptotically improves the quality of the variational approximation, but provides no guarantees on its rate of convergence or error relative to exact Bayesian inference [21].”
NGBoost: Natural Gradient Boosting for Probabilistic Prediction (2019, methodology): “Concrete Dropout improves upon MC dropout by employing a continuous relaxation of the Bernoulli distribution to automatically tune the dropout probability (Gal et al., 2017).”
What Are Bayesian Neural Network Posteriors Really Like? (2021, background): “…2018; Wu et al., 2018; Osawa et al., 2019; Dusenberry et al., 2020), dropout (Srivastava et al., 2014; Gal & Ghahramani, 2016; Kendall & Gal, 2017; Gal et al., 2017), the Laplace approximation (MacKay, 1992; Kirkpatrick et al., 2017; Ritter et al., 2018; Li, 2000; Daxberger et al., 2020),…”
Visual Attention Methods in Deep Learning: An In-Depth Survey (2022, methodology): “For this reason, they used the reparameterization trick [195],”
Structured Dialogue Policy with Graph Neural Networks (2018, methodology): “Dropout introduces some noise in the neural networks and can improve exploration in DRL (Gal and Ghahramani, 2016; Gal et al., 2017).”
Deep and Confident Prediction for Time Series at Uber (2017, methodology): “There has been recent work done on choosing the optimal dropout probability p adaptively by treating it as part of the model parameter, but this approach requires modifying the training phase [12].”
A Review and Comparative Study on Probabilistic Object Detection in Autonomous Driving (2020, methodology): “The method is later extended in [61] to ﬁnd the optimal dropout rate by treating the dropout probability as a hyperparameter.”
Optimizing Millions of Hyperparameters by Implicit Differentiation (2019, methodology): “To allow for gradient-based optimization of dropout rates, we use concrete dropout [61].”
Guided Dropout (2018, methodology): “(Gal, Hron, and Kendall 2017) have proposed “Con-crete Dropout” which is a variant of dropout where concrete distribution has been utilized to generate the dropout mask.”