Variational Dropout and the Local Reparameterization Trick

By Diederik P. Kingma, Tim Salimans, Max Welling. 2015

1,683 citations recorded by the source corpus.

We investigate a local reparameterizaton technique for greatly reducing the variance of stochastic gradients for variational Bayesian inference (SGVB) of a posterior over model parameters, while retaining parallelizability. This local reparameterization translates uncertainty about global parameters into local noise that is independent across datapoints in the minibatch. Such parameterizations can be trivially parallelized and have variance that is inversely proportional to the mini-batch size, generally leading to much faster convergence. Additionally, we explore a connection with dropout: Gaussian dropout objectives correspond to SGVB with local reparameterization, a scale-invariant prior and proportionally fixed posterior variance. Our method allows inference of more flexibly parameterized posteriors; specifically, we propose variational dropout, a generalization of Gaussian dropout where the dropout rates are learned, often leading to better models. The method is demonstrated through several experiments.

Subfields: Artificial Intelligence, Generative Models, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Generative Models

In Probabilistic & Bayesian Methods

Citations in the corpus

The State of Sparsity in Deep Neural Networks (2019, methodology): “…dropout was originally proposed as a re-interpretation of dropout training as variational inference, providing a Bayesian justiﬁcation for the use of dropout in neural networks and enabling useful extensions to the standard dropout algorithms like learnable dropout rates (Kingma et al., 2015).”
Scalable agent alignment via reward modeling: a research direction (2018, methodology): “A number of recent works develop scaleable approximate Bayesian methods for neural networks, beginning with Graves (2011), Blundell et al. (2015), Kingma et al. (2015), Hernández-Lobato & Adams (2015), and Gal & Ghahramani (2016).”
Meta-Learning Probabilistic Inference for Prediction (2018, methodology): “Standard application of amortized VI (Kingma and Welling, 2014; Rezende et al., 2014; Kingma et al., 2015; Blundell et al., 2015) for ψ in the multi-task discriminative model optimizes the Monte Carlo approximated free-energy w.r.t. φ and θ : where ψ ( t ) l ∼ q φ ( ψ D ( t ) , θ ) .”
A Theoretically Grounded Application of Dropout in Recurrent Neural Networks (2015, background): “Recent results at the intersection of Bayesian research and deep learning offer interpretation of common deep learning techniques through Bayesian eyes [11–16].”
The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks (2018, background): “Bayesian perspectives on dropout learn dropout probabilities during training (Gal et al., 2017; Kingma et al., 2015; Srinivas & Babu, 2016).”
Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles (2016, background): “There has been work on approximate Bayesian interpretation (Gal and Ghahramani, 2016; Kingma et al., 2015; Maeda, 2014) of dropout.”
Evidential Deep Learning to Quantify Classification Uncertainty (2018, methodology): “the Bayesian neural net used in [18] with the additive parametrization [26], (e) MNFG refers to the structured variational inference method used in [24], (f) EDL is the method we propose.”
Decentralized Bayesian Learning over Graphs (2019, positive): “Gaussian approximate posterior can be obtained in an computationally efﬁcient manner via VI techniques [10, 5].”
A Comprehensive guide to Bayesian Convolutional Neural Network with Variational Inference (2019, methodology): “Several authors have claimed that Dropout [58] and Gaussian Dropout [61] can be viewed as approximate variational inference schemes [13, 32].”
Learning Sparse Neural Networks through L0 Regularization (2017, methodology): “We used a single sample of the gate z for each minibatch of datapoints during the optimization, even though this can lead to larger variance in the gradients (Kingma et al., 2015).”
Physics-Constrained Deep Learning for High-dimensional Surrogate Modeling and Uncertainty Quantification without Labeled Data (2019, methodology): “variational dropout [16, 17], Stein variational gradient descent [18, 9], although other methods exist, e.”
Bayesian Hypernetworks (2017, background): “Notable recent work in this area includes Gal and Ghahramani (2016) and Kingma et al. (2015), who interpret the popular dropout (Srivastava et al., 2014) algorithm as a variational inference method ("MC dropout").”
Critical Learning Periods in Deep Neural Networks (2017, methodology): “Notice that L′ can be minimized efficiently using the method in (58).”
Automated Curriculum Learning for Neural Networks (2017, background): “MDL training for neural networks (Hinton and Van Camp, 1993) can be practically realised with stochastic variational inference (Graves, 2011; Kingma et al., 2015; Blundell et al., 2015).”
Posterior Meta-Replay for Continual Learning (2021, methodology): “[30] D.”
Continual Learning with Adaptive Weights (CLAW) (2019, methodology): “The switch from (10) to (11) is due to using a log-scale prior, similar to Appendix C in (Kingma et al., 2015) and to Section 4.”
Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networks (2019, methodology): “Since the noise is Gaussian, L(Σ) can be optimized using the local reparametrization trick [16].”
Improving and Understanding Variational Continual Learning (2019, methodology): “We also ﬁrstly introduce the local reparameterisation trick during the Monte Carlo sampling of the network [9], and secondly initialise all weights with small, random means (of the order 10 − 1 ) and small variances (of the order 10 − 3 ) before every task.”
Multiplicative Normalizing Flows for Variational Bayesian Neural Networks (2017, methodology): “One avenue would be to explore how much can MNFs sparsify and compress neural networks under either sparsity inducing priors, such as the log-uniform prior (Kingma et al., 2015; Molchanov et al., 2017), or empirical priors (Ullrich et al., 2017).”