Soft Weight-Sharing for Neural Network Compression

By Karen Ullrich, Edward Meeds, Max Welling. 2017

444 citations recorded by the source corpus.

The success of deep learning in numerous application domains created the desire to run and train them on mobile devices. This however, conflicts with their computationally, memory and energy intense nature, leading to a growing interest in compression. Recent work by Han et al. (2015a) propose a pipeline that involves retraining, pruning and quantization of neural network weights, obtaining state-of-the-art compression rates. In this paper, we show that competitive compression rates can be achieved by using a version of “soft weight-sharing” (Nowlan & Hinton, 1992). Our method achieves both quantization and pruning in one simple (re-)training procedure. This point of view also exposes the relation between compression and the minimum description length (MDL) principle.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

The State of Sparsity in Deep Neural Networks (2019, methodology): “Many techniques grounded in Bayesian statistics and information theory have been proposed (Dai et al., 2018; Molchanov et al., 2017; Louizos et al., 2017b;a; Ullrich et al., 2017).”
Controlled Sparsity via Constrained Optimization or: How I Learned to Stop Tuning Penalties and Love Constraints (2022, background): “Commonly used neural networks result in overparametrized models, whose performance is robust to harsh levels of parameter pruning [18, 42, 12, 14].”
BayesNAS: A Bayesian Approach for Neural Architecture Search (2019, methodology): “Other Bayesian methods have also been applied to network pruning (Ullrich et al., 2017; Molchanov et al., 2017a) where the former extends the soft weight-sharing to obtain a sparse and compressed network and the latter uses variational inference to learn the dropout rate that can then be used for…”
Learning Sparse Neural Networks through L0 Regularization (2017, background): “Firstly, it has been shown in recent works (Han et al., 2015; Ullrich et al., 2017; Molchanov et al., 2017) that they are greatly overparametrized as they can be pruned significantly without any loss in accuracy; this exhibits unnecessary computation and resources.”
Soft-to-Hard Vector Quantization for End-to-End Learning Compressible Representations (2017, methodology): “A notable different approach is taken by [34], where the DNN”
Differentiable Model Compression via Pseudo Quantization Noise (2021, methodology): “Finally Ullrich, Meeds, and Welling (2017) uses a gaussian mixture model trained on top of the weights, adding significant complexity both in terms of code, and computation.”
Multiplicative Normalizing Flows for Variational Bayesian Neural Networks (2017, background): “One avenue would be to explore how much can MNFs sparsify and compress neural networks under either sparsity inducing priors, such as the log-uniform prior (Kingma et al., 2015; Molchanov et al., 2017), or empirical priors (Ullrich et al., 2017).”
Variational Dropout Sparsifies Deep Neural Networks (2017, methodology): “We consider pruning (Han et al., 2015b;a), Dynamic Network Surgery (Guo et al., 2016) and Soft Weight Sharing (Ullrich et al., 2017).”
Bayesian Compression for Deep Learning (2017, methodology): “There have been quite a few encouraging results obtained by (empirical) Bayesian approaches that employ weight pruning [19, 7, 52, 70, 51].”
The Description Length of Deep Learning models (2018, methodology): “, 2015), or pipelines involving retraining, pruning, and quantization of the model weights (Han et al., 2015a,b; Simonyan and Zisserman, 2014; Louizos et al., 2017; See et al., 2016; Ullrich et al., 2017).”
Sparse Networks from Scratch: Faster Training without Losing Performance (2019, methodology): “t loss/pruning-cost optimization (Carreira-Perpinán and Idelbayev, 2018), layer-by-layer pruning (Dong et al., 2017), fast-switching growth-pruning cycles (Dai et al., 2017), and soft weight-sharing (Ullrich et al., 2017). These approaches often involve re-training phases which increase the training”
SNIP: Single-shot Network Pruning based on Connection Sensitivity (2018, methodology): “…our approach for extreme sparsity levels ( e.g ., up to 99 % sparsity on LeNet-5-Caffe ) and compare ours against the following various pruning algorithms: LWC (Han et al. (2015)), DNS (Guo et al. (2016 2017)), SVD (Molchanov et al. (2017)), OBD (LeCun et al. (1990)), L-OBS (Dong et al. (2017)).”
Structured Bayesian Pruning via Log-Normal Multiplicative Noise (2017, methodology): “High sparsity is one of the key factors for the compression of DNNs [5, 21].”
Model Compression by Entropy Penalized Reparameterization (2019, methodology): “Other methods explore vector quantization (VQ), which is closely related to k-means clustering, in which each vector of filter coefficients is quantized jointly (Chen et al., 2015; Ullrich et al., 2017).”
Entropy-Constrained Training of Deep Neural Networks (2018, background): “Therefore, [22], [34], [37] focused on designing suitable priors and posteriors that allow to apply practical coding schemes on the weight parameters after the variational lower bound has been minimized.”
Soft-to-Hard Vector Quantization for End-to-End Learned Compression of Images and Neural Networks (2017, methodology): “…stream to be i.i.d. and to model the marginal symbol distribution with a parametric model, such as a Gaussian mixture model (Theis et al., 2017; Ullrich et al., 2017), a piecewise linear model (Ballé et al., 2016b), or a Bernoulli distribution (Toderici et al., 2016) (in the case of binary…”
HiLLoC: Lossless Image Compression with Hierarchical Latent Variable Models (2019, background): “This is a well-established research area in machine learning (Han et al., 2016; Ullrich et al., 2017).”
Frequency-Domain Dynamic Pruning for Convolutional Neural Networks (2018, methodology): “[16] revived a principled regularization method based on soft weight-sharing.”
Accelerating Convolutional Neural Networks via Activation Map Compression (2018, methodology): “Pruning of weights [14, 18, 19, 34, 36, 40, 48, 63, 69] and quantization of weights and activations [6, 7, 12, 15, 18, 23, 50, 64, 67] are the standard compression techniques that are currently used.”