Bayesian Compression for Deep Learning

By Christos Louizos, Karen Ullrich, Max Welling. 2017

508 citations recorded by the source corpus.

Compression and computational efficiency in deep learning have become a problem of great significance. In this work, we argue that the most principled and effective way to attack this problem is by adopting a Bayesian point of view, where through sparsity inducing priors we prune large parts of the network. We introduce two novelties in this paper: 1) we use hierarchical priors to prune nodes instead of individual weights, and 2) we use the posterior uncertainties to determine the optimal fixed point precision to encode the weights. Both factors significantly contribute to achieving the state of the art in terms of compression rates, while still staying competitive with methods designed to optimize for speed or energy efficiency.

Subfields: Artificial Intelligence, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Efficiency & Compression

Citations in the corpus

The State of Sparsity in Deep Neural Networks (2019, methodology): “At test-time, Louizos et al. (2017b) use the following estimate for the model parameters.”
Measuring the Intrinsic Dimension of Objective Landscapes (2018, methodology): “( iii ) Compared to methods like that of Louizos et al. (2017), who take a Bayesian perspective and consider redundancy on the level of groups of parameters (input weights to a single neuron) by using group-sparsity-inducing hierarchical priors on the weights, our approach is simpler but not likely…”
The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks (2018, background): “, 2017) or explicitly (Louizos et al., 2017; Srinivas & Babu, 2015b) prune and sparsify networks during training as dropout probabilities for some weights reach 1.”
What is the State of Neural Network Pruning? (2020, methodology): “…many methods produce pruned models that outperform re-training from scratch with the same sparsity pattern (Zhang et al., 2015; Yu et al., 2018; Louizos et al., 2017; He et al., 2017; Luo et al., 2017; Frankle & Carbin, 2019) (at least with a large enough amount of pruning (Suau et al., 2018)).”
Split learning for health: Distributed deep learning without sharing raw patient data (2018, background): “This uneven distri-3 [36, 37, 38] for seamless distributed learning with edge devices.”
BayesNAS: A Bayesian Approach for Neural Architecture Search (2019, methodology): “However, it can be different for W or constructed to represent the structural sparsity for Convolutional kernels in NN aiming for Network Compression, which is related to Bayesian compression (Louizos et al., 2017) and structural sparsity compression (Wen et al.”
Layer-compensated Pruning for Resource-constrained Convolutional Neural Networks (2018, methodology): “We note that among the prior methods that we are comparing, ﬁve of them (Molchanov et al., 2017; Gordon et al., 2018; He et al., 2018b; Louizos et al., 2017; Dai et al., 2018) target the layer scheduling problem directly or indirectly while others focus on the complementary problems such as the…”
Learning Sparse Neural Networks through L0 Regularization (2017, methodology): “…1.8 266-88-33 1.8 LeNet-5-Caffe Sparse VD (Molchanov et al., 2017) 14-19-242-131 1.0 20-50-800-500 GL (Wen et al., 2016) 3-12-192-500 1.0 SBP (Neklyudov et al., 2017) 3-18-284-283 0.9 BC-GNJ (Louizos et al., 2017) 8-13-88-13 1.0 BC-GHS (Louizos et al., 2017) 5-10-76-16 1.0 L 0 hc , λ = 0 .”
Conditional Neural Processes (2018, background): “Finally, the latent variant of CNP can also be seen as an approximated amortized version of Bayesian DL (Gal & Ghahramani, 2016; Blundell et al., 2015; Louizos et al., 2017; Louizos & Welling, 2017)”
Continual Learning with Adaptive Weights (CLAW) (2019, methodology): “To automate adaptation, we perform inference on pi,j, which would have otherwise been a hyperparameter of the prior (Louizos et al., 2017; Molchanov et al., 2017; Ghosh et al., 2018).”
Improving and Understanding Variational Continual Learning (2019, methodology): “Previous work on Variational Inference for BNNs has found that VI methods can be used to prune large parts of the network [12], and we show here how this pruning is done over units as opposed to weights (see also Trippe and Turner [20]).”
Neural Processes (2018, methodology): “…model in Figure 2d we can write out the conditional marginal likelihood as a hierarchical inference problem: Another interesting area that connects Bayesian methods and NNs are Bayesian neural networks (Gal & Ghahramani, 2016; Blundell et al., 2015; Louizos et al., 2017; Louizos & Welling, 2017).”
Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors (2020, methodology): “It has long been conjectured that such priors can be more robust to out-of-distribution data while inducing sparsity (Louizos et al., 2017) at the expense of accuracy.”
Reliable Uncertainty Estimates in Deep Neural Networks using Noise Contrastive Priors (2018, methodology): “Recent works analyzing priors have been specific to compression and model selection (Ghosh and Doshi-Velez, 2017; Louizos et al., 2017); this is with the exception of Flam-Shepherd et al.”
Selfless Sequential Learning (2018, methodology): “Differently from most of the compression based approaches [26, 1, 27], we argue that achieving sparsity in the network parameters is less significant in a sequential learning scenario than sparsity in the neural activations.”
Information-Theoretic Probing with Minimum Description Length (2020, methodology): “With variational probes, we do not anneal learning rate and train probes for 200 epochs; long training is recommended to enable pruning (Louizos et al., 2017).”
Flipout: Efficient Pseudo-Independent Weight Perturbations on Mini-Batches (2018, background): “Control variates are another general class of strategies for variance reduction, both for black-box optimization (Williams, 1992; Ranganath et al., 2014; Mnih & Gregor, 2014) and for gradient-based optimization (Roeder et al., 2016; Miller et al., 2017; Louizos et al., 2017).”