Structured Bayesian Pruning via Log-Normal Multiplicative Noise

By Kirill Neklyudov, Dmitry Molchanov, Arsenii Ashukha, Dmitry P. Vetrov. 2017

193 citations recorded by the source corpus.

Dropout-based regularization methods can be regarded as injecting random noise with pre-defined magnitude to different parts of the neural network during training. It was recently shown that Bayesian dropout procedure not only improves generalization but also leads to extremely sparse neural architectures by automatically setting the individual noise magnitude per weight. However, this sparsity can hardly be used for acceleration since it is unstructured. In the paper, we propose a new Bayesian model that takes into account the computational structure of neural networks and provides structured sparsity, e.g. removes neurons and/or convolutional channels in CNNs. To do this we inject noise to the neurons outputs while keeping the weights unregularized. We establish the probabilistic model with a proper truncated log-uniform prior over the noise and truncated log-normal variational approximation that ensures that the KL-term in the evidence lower bound is computed in closed-form. The model leads to structured sparsity by removing elements with a low SNR from the computation graph and provides significant acceleration on a number of deep neural architectures. The model is easy to implement as it can be formulated as a separate dropout-like layer.

Subfields: Artificial Intelligence, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Efficiency & Compression

Citations in the corpus

Controlled Sparsity via Constrained Optimization or: How I Learned to Stop Tuning Penalties and Love Constraints (2022, methodology): “Structured pruning methods [6, 37, 31, 29], remove entire neurons/channels rather than individual parameters.”
The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks (2018, background): “Techniques that learn perweight, per-unit (Srinivas & Babu, 2016), or structured dropout probabilities naturally (Molchanov et al., 2017; Neklyudov et al., 2017) or explicitly (Louizos et al.”
Dynamic Model Pruning with Feedback (2020, background): “Gal et al. (2017); Neklyudov et al. (2017); Molchanov et al. (2017) prune from Bayesian perspectives to learn dropout probabilities during training to prune and sparsify networks as dropout weight probabilities reach 1.”
BayesNAS: A Bayesian Approach for Neural Architecture Search (2019, neutral): “FLOPs (M) Baseline 784-300-100 1.39 0.53 SBP ((Neklyudov et al., 2017)) 245-160-55 1.60 0.10 BC-GNJ ((Louizos et al., 2017)) 278-98-13 1.80 0.06 BC-GHS ((Louizos et al., 2017)) 311-86-14 1.80 0.06 Practical `0((Louizos et al., 2018)) 219-214-100 1.40 0.14 Practical `0 ((Louizos et al., 2018))…”
Learning Sparse Neural Networks through L0 Regularization (2017, methodology): “…1.8 266-88-33 1.8 LeNet-5-Caffe Sparse VD (Molchanov et al., 2017) 14-19-242-131 1.0 20-50-800-500 GL (Wen et al., 2016) 3-12-192-500 1.0 SBP (Neklyudov et al., 2017) 3-18-284-283 0.9 BC-GNJ (Louizos et al., 2017) 8-13-88-13 1.0 BC-GHS (Louizos et al., 2017) 5-10-76-16 1.0 L 0 hc , λ = 0 .”
Rigging the Lottery: Making All Tickets Winners (2019, methodology): “In this section we compare RigL with three different structured pruning algorithms: SBP (Neklyudov et al., 2017), L0 (Christos Louizos, 2018), and VIB (Dai et al., 2018).”
Modular Networks: Learning to Decompose Neural Computation (2018, methodology): “ation sparsity or winner-take-all mechanisms [27, 23, 24] but is hard to parallelize on modern accelerators such as GPUs. A solution that works with these accelerators is learning structured sparsity [18, 29] but often requires non-sparse computation during training or is not conditional. 5 Conclusi”
Importance Estimation for Neural Network Pruning (2019, methodology): “Therefore, prior work has tried to relax the optimization using Bayesian methods [25, 29] or regularization terms.”
Emergence of Invariance and Disentanglement in Deep Representations (2017, methodology): “…with this choice the information in the weights I ( w ; D ) can be expressed in closed form, up to an arbitrary constant C which does not matter during the optimization process (but see also Neklyudov et al. (2017) for a principled approach to this problem that uses a proper log-uniform prior).”
Compressing Neural Networks using the Variational Information Bottleneck (2018, methodology): “For this purpose, multiple interesting Bayesian approaches have been proposed (Louizos et al., 2017a; Neklyudov et al., 2017) that rely on sparse priors (e.”
Energy-Constrained Compression for Deep Neural Networks via Weighted Sparse Projection and Layer Input Masking (2018, methodology): “We compare our method mainly with ﬁve state-of-art pruning methods: magnitude-based pruning (MP) (Han et al., 2015b;a), structured sparsity learning (SSL) (Wen et al., 2016), structured bayesian pruning (SBP) (Neklyudov et al., 2017), bayesian compression (BC) (Louizos et al., 2017) and energy-aware”
Taxonomy and Evaluation of Structured Compression of Convolutional Neural Networks (2019, methodology): “The methods from Louizos et al. (2017) and Neklyudov et al. (2017) are variants of this method with different regularization functions F .”
Variance Networks: When Expectation Does Not Meet Your Expectations (2018, methodology): “• Previous works used the signal-to-noise ratio of the weights or the layer output to prune excessive units [2, 22, 24].”
Frequency-Domain Dynamic Pruning for Convolutional Neural Networks (2018, methodology): “[21], [22] and [23] prune nodes or filters by employing Bayesian point of view and L0 norm regularization.”
Compressing RNNs for IoT devices by 15-38x using Kronecker Products (2019, methodology): “While there are multiple possible ways to prune [34, 37], magnitude pruning has shown comparable or better accuracy”
The Lottery Ticket Hypothesis: Training Pruned Neural Networks (2018, background): “ possible in the training process or to integrate lessons from winning tickets directly into the design of new architectures and initialization schemes. Techniques that prune networks during training [35, 38, 41] might already identify winning tickets and could beneﬁt from making this an explicit go”
Emergence of invariance and disentangling in deep representations (2017, methodology): “Neklyudov et al. (2017) also suggest a principled way of dealing with the arbitrary constant by using a proper log-uniform prior.”