Variational Convolutional Neural Network Pruning

By Chenglong Zhao, Bingbing Ni, Jian Zhang, Qiwei Zhao, Wenjun Zhang, Qi Tian. Computer Vision and Pattern Recognition, 2019

350 citations recorded by the source corpus.

We propose a variational Bayesian scheme for pruning convolutional neural networks in channel level. This idea is motivated by the fact that deterministic value based pruning methods are inherently improper and unstable. In a nutshell, variational technique is introduced to estimate distribution of a newly proposed parameter, called channel saliency, based on this, redundant channels can be removed from model via a simple criterion. The advantages are two-fold: 1) Our method conducts channel pruning without desire of re-training stage, thus improving the computation efficiency. 2) Our method is implemented as a stand-alone module, called variational pruning layer, which can be straightforwardly inserted into off-the-shelf deep learning packages, without any special network design. Extensive experimental results well demonstrate the effectiveness of our method: For CIFAR-10, we perform channel removal on different CNN models up to 74\% reduction, which results in significant size reduction and computation saving. For ImageNet, about 40% channels of ResNet-50 are removed without compromising accuracy.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/b39eec4d962bb1cae0156af6d7f34c1b6302dc51

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Kurt Keutzer
2. A. Gholami
3. Michael W. Mahoney
4. Zi-Mei Yao
5. Sehoon Kim
6. Sheng Shen
7. Yichen Zhang
8. Linjian Ma

In Computer Vision and Pattern Recognition

1. Kurt Keutzer
2. A. Gholami
3. Michael W. Mahoney
4. Zi-Mei Yao
5. Sehoon Kim
6. Sheng Shen
7. Ling Shao
8. Yang He

Citations in the corpus

• MLPruning: A Multilevel Structured Pruning Framework for Transformer-based Models (2021, background): “Pruning can be broadly categorized into unstructured pruning [9, 24, 49, 29, 15, 36] and structured pruning [26, 16, 51, 25, 18, 56, 52, 27].”
• A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “Pruning methods can be broadly categorized into unstructured pruning [49, 85, 137, 141, 186, 249], and structured pruning [90, 105, 153, 162, 266, 267, 271].”
• HRank: Filter Pruning Using High-Rank Feature Map (2020, background): “Moreover, compared with existing property importance based methods, HRank also leads to significant improvements in acceleration and compression [18, 34] , achieving even better results than state-of-the-art adaptive importance based methods [13, 24, 26, 16, 23, 36].”
• Learned Token Pruning for Transformers (2021, methodology): “Pruning methods can be further categorized into unstructured pruning [12, 28, 33, 39, 54] and structured pruning [17, 18, 20, 29, 31, 49, 56, 57, 62].”
• Structured Pruning for Deep Convolutional Neural Networks: A Survey (2023, background): “Variational Pruning (VP) [102] is based on channel importance being indicated by random variables, as pruning by deterministic channel importance is inherently improper and unstable.”
• NeuralScale: Efficient Scaling of Neurons for Resource-Constrained Deep Neural Networks (2020, methodology): “There are two ways to prune a network: structured pruning [32, 20, 39, 28, 34, 52] and unstructured pruning [13, 14, 15, 9].”