Accelerating Convolutional Networks via Global & Dynamic Filter Pruning

By Shaohui Lin, Rongrong Ji, Yuchao Li, Yongjian Wu, Feiyue Huang, Baochang Zhang. 2018

271 citations recorded by the source corpus.

Accelerating convolutional neural networks has recently received ever-increasing research focus. Among various approaches proposed in the literature, filter pruning has been regarded as a promising solution, which is due to its advantage in significant speedup and memory reduction of both network model and intermediate feature maps. To this end, most approaches tend to prune filters in a layer-wise fixed manner, which is incapable to dynamically recover the previously removed filter, as well as jointly optimize the pruned network across layers. In this paper, we propose a novel global & dynamic pruning (GDP) scheme to prune redundant filters for CNN acceleration. In particular, GDP first globally prunes the unsalient filters across all layers by proposing a global discriminative function based on prior knowledge of filters. Second, it dynamically updates the filter saliency all over the pruned sparse network, and then recover the mistakenly pruned filter, followed by a retraining phase to improve the model accuracy. Specially, we effectively solve the corresponding non-convex optimization problem of the proposed GDP via stochastic gradient descent with greedy alternative updating. Extensive experiments show that, comparing to the state-of-the-art filter pruning methods, the proposed approach achieves superior performance to accelerate several cutting-edge CNNs on the ILSVRC 2012 benchmark.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Kurt Keutzer
2. Michael W. Mahoney
3. A. Gholami
4. Zi-Mei Yao
5. Yunhe Wang
6. Dacheng Tao
7. Qi Tian
8. Sheng Shen

Citations in the corpus

• A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “Pruning methods can be broadly categorized into unstructured pruning [49, 84, 135, 139, 184, 244], and structured pruning [89, 104, 151, 160, 261, 262, 267].”
• DeepHoyer: Learning Sparser Neural Network with Differentiable Scale-Invariant Sparsity Measures (2019, unknown): “05 × GDP (Lin et al., 2018) 71.89% 90.71% 2 .”
• Data-Independent Structured Pruning of Neural Networks via Coresets (2020, methodology): “The method in [37] first prunes low saliency filters across all layers, then dynamically updates the filter saliency, and recovers the mistakenly pruned filter.”
• Learned Token Pruning for Transformers (2021, methodology): “Pruning methods can be further categorized into unstructured pruning [12, 28, 33, 39, 54] and structured pruning [17, 18, 20, 29, 31, 49, 56, 57, 62].”
• Discrimination-Aware Network Pruning for Deep Model Compression (2020, methodology): “On ILSVRC-12, we compare DCP with SFP [24], SPP [75], CP [27], GDP [43], FPGM [26], ThiNet [51], SSR-L2 [42], AMC [25], and MetaPruning [48].”
• Group Fisher Pruning for Practical Network Compression (2021, unknown): “…76.18 2.25 - C-SGD (Ding et al., 2019a) 74.54 75.33 2.20 - Meta (Liu et al., 2019) 75.40 76.60 2.0 - GBN (You et al., 2019) 75.18 75.85 1.84 - LFPC (He et al., 2020) 74.46 76.15 1.60 - HRank (Lin et al., 2020) 74 (Luo et al., 2017) 68.17 75.30 1.17 1.35 GDP (Lin et al., 2018) 70.93 75.13 1.57 -”
• Dynamic Sparse Training: Find Efficient Sparse Network From Scratch With Trainable Masked Layers (2020, methodology): “For the dynamic strategy, a single global pruning threshold (Molchanov et al. (2019); Lin et al. (2018)) or layer-wise greedy algorithms (Luo et al. (2017); He et al. (2017)) are applied to achieve different layer-wise sparsity.”
• Towards Optimal Structured CNN Pruning via Generative Adversarial Learning (2019, methodology): “For instance, Lin et al. [35] learned a global mask with binary values to determine the saliency of filters, and pruned the redundant filters by masking out the corresponding mask as 0.”
• HRank: Filter Pruning Using High-Rank Feature Map (2020, unknown): “6 [22] 71.”
• MLPruning: A Multilevel Structured Pruning Framework for Transformer-based Models (2021, background): “Pruning can be broadly categorized into unstructured pruning [9, 24, 49, 29, 15, 36] and structured pruning [26, 16, 51, 25, 18, 56, 52, 27].”
• CARS: Continuous Evolution for Efficient Neural Architecture Search (2019, methodology): “The efficient block design [54, 53, 52], tensor decomposition [24, 49, 50, 42, 43], pruning [25, 26, 12], distillation [4, 48] and quantization [22] are popular techniques to make networks efficient.”
• Gate Decorator: Global Filter Pruning Method for Accelerating Deep Convolutional Neural Networks (2019, unknown): “45 GDP [26] 3 71.”