Gate Decorator: Global Filter Pruning Method for Accelerating Deep Convolutional Neural Networks

By Zhonghui You, Kun Yan, Jinmian Ye, Meng Ma, Ping Wang. 2019

295 citations recorded by the source corpus.

Filter pruning is one of the most effective ways to accelerate and compress convolutional neural networks (CNNs). In this work, we propose a global filter pruning algorithm called Gate Decorator, which transforms a vanilla CNN module by multiplying its output by the channel-wise scaling factors (i.e. gate). When the scaling factor is set to zero, it is equivalent to removing the corresponding filter. We use Taylor expansion to estimate the change in the loss function caused by setting the scaling factor to zero and use the estimation for the global filter importance ranking. Then we prune the network by removing those unimportant filters. After pruning, we merge all the scaling factors into its original module, so no special operations or structures are introduced. Moreover, we propose an iterative pruning framework called Tick-Tock to improve pruning accuracy. The extensive experiments demonstrate the effectiveness of our approaches. For example, we achieve the state-of-the-art pruning ratio on ResNet-56 by reducing 70% FLOPs without noticeable loss in accuracy. For ResNet-50 on ImageNet, our pruned model with 40% FLOPs reduction outperforms the baseline model by 0.31% in top-1 accuracy. Various datasets are used, including CIFAR-10, CIFAR-100, CUB-200, ImageNet ILSVRC-12 and PASCAL VOC 2011.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Dynamic Model Pruning with Feedback (2020, background): “through `0 regularization) that minimize the number of nonzero weights, recent parallel work studies filter pruning for pre-trained models (You et al., 2019).”
Group Fisher Pruning for Practical Network Compression (2021, methodology): “…1.90 1.62 IE (Molchanov et al., 2019) 74.50 76.18 2.25 - C-SGD (Ding et al., 2019a) 74.54 75.33 2.20 - Meta (Liu et al., 2019) 75.40 76.60 2.0 - GBN (You et al., 2019) 75.18 75.85 1.84 - LFPC (He et al., 2020) 74.46 76.15 1.60 - HRank (Lin et al., 2020) 74 (Luo et al., 2017) 68.17 75.30 1.17 1.35…”
A Survey on Deep Neural Network Pruning: Taxonomy, Comparison, Analysis, and Recommendations (2023, unknown): “The ﬁrst two types correspond to spe-ciﬁc speedup, while the third corresponds to universal speedup.”
DepGraph: Towards Any Structural Pruning (2023, methodology): “Mainstream pruning methods can be broadly categorized into two types: structural pruning [4,29,32,71,71] and unstructural pruning [8,28,44,49].”
Data-Independent Structured Pruning of Neural Networks via Coresets (2020, background): “3088587 have been proposed (see [15] for a recent review on network sparsification and [22], [23], [36], [43], and [55]–[57] for structured pruning).”
Structured Pruning for Deep Convolutional Neural Networks: A Survey (2023, methodology): “SimilartothesituationinNS[74],GatedBatchNormalization (GBN) [75] uses γ as the channel-wise gate and (cid”
Efficient Acceleration of Deep Learning Inference on Resource-Constrained Edge Devices: A Review (2023, methodology): “Gate decorator [106] is a global filter pruning algorithm for CNN based on scaling the channels.”
Discrimination-Aware Network Pruning for Deep Model Compression (2020, background): “In order to measure the importance of network module, different metrics [26], [29], [40], [41], [58], [80] are proposed.”
Channel Selection Using Gumbel Softmax (2020, methodology): “Contemporary with our work, [52] uses a Taylor expansion rather than Gumbel to estimate the impact of opening or closing a gate; their technique prunes the network, but has no natural extension to conditional computation.”
Learning Dynamic Routing for Semantic Segmentation (2020, methodology): “Traditional methods mainly focus on image classification by dropping blocks [38, 17, 33, 36] or pruning channels [39, 20] for efficient inference.”