Auto-Balanced Filter Pruning for Efficient Convolutional Neural Networks

By Xiaohan Ding, Guiguang Ding, Jungong Han, Sheng Tang. 2018

127 citations recorded by the source corpus.

In recent years considerable research efforts have been devoted to compression techniques of convolutional neural networks (CNNs). Many works so far have focused on CNN connection pruning methods which produce sparse parameter tensors in convolutional or fully-connected layers. It has been demonstrated in several studies that even simple methods can effectively eliminate connections of a CNN. However, since these methods make parameter tensors just sparser but no smaller, the compression may not transfer directly to acceleration without support from specially designed hardware. In this paper, we propose an iterative approach named Auto-balanced Filter Pruning, where we pre-train the network in an innovative auto-balanced way to transfer the representational capacity of its convolutional layers to a fraction of the filters, prune the redundant ones, then re-train it to restore the accuracy. In this way, a smaller version of the original network is learned and the floating-point operations (FLOPs) are reduced. By applying this method on several common CNNs, we show that a large portion of the filters can be discarded without obvious accuracy drop, leading to significant reduction of computational burdens. Concretely, we reduce the inference cost of LeNet-5 on MNIST, VGG-16 and ResNet-56 on CIFAR-10 by 95.1%, 79.7% and 60.9%, respectively.

Subfields: Computational Mathematics, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mathematics

In Computer Vision and Pattern Recognition

Citations in the corpus

What is the State of Neural Network Pruning? (2020, background): “, 2019b), or added dropout or batchnorm (Liu et al., 2017; Lee et al., 2019b; Peng et al., 2018; Molchanov et al., 2017; Ding et al., 2018; Suau et al., 2018).”
RepVGG: Making VGG-style ConvNets Great Again (2021, background): “However, multibranch models make pruning tricky and result in significant performance degradation or low acceleration ratio [7, 22, 9].”
Neural Pruning via Growing Regularization (2020, methodology): “To resolve this, Ding et al. (2018) and Wang et al.”
ACNet: Strengthening the Kernel Skeletons for Powerful CNN via Asymmetric Convolution Blocks (2019, background): “, absolute value) as the metric for the importance of parameters, which is adopted by many prior CNN pruning works [7, 9, 12, 21].”
A Survey on Deep Neural Network Pruning: Taxonomy, Comparison, Analysis, and Recommendations (2023, unknown): “Considering the huge search space of network encoding vectors, the pruned network is found by evolutionary search under the constraints.”