Variational Dropout Sparsifies Deep Neural Networks

By Dmitry Molchanov, Arsenii Ashukha, Dmitry P. Vetrov. 2017

897 citations recorded by the source corpus.

We explore a recently proposed Variational Dropout technique that provided an elegant Bayesian interpretation to Gaussian Dropout. We extend Variational Dropout to the case when dropout rates are unbounded, propose a way to reduce the variance of the gradient estimator and report first experimental results with individual dropout rates per weight. Interestingly, it leads to extremely sparse solutions both in fully-connected and convolutional layers. This effect is similar to automatic relevance determination effect in empirical Bayes but has a number of advantages. We reduce the number of parameters up to 280 times on LeNet architectures and up to 68 times on VGG-like networks with a negligible decrease of accuracy.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

The State of Sparsity in Deep Neural Networks (2019, methodology): “Our main contributions: (1) We perform a comprehensive evaluation of variational dropout (Molchanov et al., 2017), l0 regularization (Louizos et al.”
Controlled Sparsity via Constrained Optimization or: How I Learned to Stop Tuning Penalties and Love Constraints (2022, methodology): “Popular non-magnitude based techniques include [26, 17, 34].”
The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks (2018, background): “, 2013) or learn per-weight dropout probabilities (Molchanov et al., 2017; Louizos et al., 2018) could make winning tickets even easier to find.”
Evidential Deep Learning to Quantify Classification Uncertainty (2018, methodology): “Improvement of the approximation techniques, such as Variational Bayes (VB) [2, 6, 27, 23, 9] and Stochastic Gradient Hamiltonian Monte Carlo (SG-HMC) [3] tailored specifically for scalable inference of BNNs is an active research field.”
What is the State of Neural Network Pruning? (2020, background): “Others modify the network to explicitly include additional parameters that encourage sparsity and serve as a basis for scoring the network after training (Molchanov et al., 2017).”
A Comprehensive guide to Bayesian Convolutional Neural Network with Variational Inference (2019, background): “As this formulation of the variance includes the mean μijwh, only αijhw needs to be learned in the second convolutional operation [47].”
Dynamic Model Pruning with Feedback (2020, background): “Gale et al. (2019) extensively study recent unstructured pruning methods on large-scale learning tasks, and find that complex techniques (Molchanov et al., 2017; Louizos et al., 2018) perform inconsistently.”
BayesNAS: A Bayesian Approach for Neural Architecture Search (2019, methodology): “The learned structure is 5− 10− 65− 25 with 1.00% test error and 0.57 FLOPS.”
Learning Sparse Neural Networks through L0 Regularization (2017, methodology): “Network & size Method Pruned architecture Error (%) MLP Sparse VD (Molchanov et al., 2017) 512-114-72 1.”
Generalized Variational Continual Learning (2020, background): “In the single task setting, this is often relatively benign and can even facilitate compression (Louizos et al., 2017; Molchanov et al., 2017).”
Continual Learning with Adaptive Weights (CLAW) (2019, methodology): “Eq(w|γ) log | | is computed via an accurate approximation similar to equation (14) in (Molchanov et al., 2017), with slightly different values of k1, k2 and k3.”
Multiplicative Normalizing Flows for Variational Bayesian Neural Networks (2017, methodology): “One avenue would be to explore how much can MNFs sparsify and compress neural networks under either sparsity inducing priors, such as the log-uniform prior (Kingma et al., 2015; Molchanov et al., 2017), or empirical priors (Ullrich et al., 2017).”
Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors (2020, background): “…for variational BNNs. Sampling with rank-1 factors (Equation 1) is closely re-lated to Gaussian local reparameterization (Kingma et al., 2015; Molchanov et al., 2017), where noise is reparameter-ized to act on the hidden units to enable weight sampling per-example, providing signiﬁcant…”
Bayesian Compression for Deep Learning (2017, methodology): “It should be mentioned that this prior parametrization readily allows for a more flexible marginal posterior over the weights as we now have a compound distribution, qφ(W) = ∫ qφ(W|z)qφ(z)dz; this is in contrast to the original parametrization and the Gaussian approximations employed by [37, 51].”
Rethinking the Value of Network Pruning (2018, methodology): “Unstructured pruning (Han et al., 2015; Molchanov et al., 2017; Louizos et al., 2018) also falls in the category of automatic methods, where the positions of pruned weights are determined by the training process and the pruning algorithm, and it is usually not possible to predefine the positions of ”
Soft Weight-Sharing for Neural Network Compression (2017, background): “A variational version is is proposed by Molchanov et al. (2017), the authors learn the dropout rate for each weight in the network separately.”