Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Acceleration

By Yang He, Ping Liu, Ziwei Wang, Zhilan Hu, Yi Yang. 2018

1,252 citations recorded by the source corpus.

Previous works utilized “smaller-norm-less-important” criterion to prune filters with smaller norm values in a convolutional neural network. In this paper, we analyze this norm-based criterion and point out that its effectiveness depends on two requirements that are not always met: (1) the norm deviation of the filters should be large; (2) the minimum norm of the filters should be small. To solve this problem, we propose a novel filter pruning method, namely Filter Pruning via Geometric Median (FPGM), to compress the model regardless of those two requirements. Unlike previous methods, FPGM compresses CNN models by pruning filters with redundancy, rather than those with“relatively less” importance. When applied to two image classification benchmarks, our method validates its usefulness and strengths. Notably, on CIFAR-10, FPGM reduces more than 52% FLOPs on ResNet-110 with even 2.69% relative accuracy improvement. Moreover, on ILSVRC-2012, FPGM reduces more than 42% FLOPs on ResNet-101 without top-5 accuracy drop, which has advanced the state-of-the-art. Code is publicly available on GitHub: https://github.com/he-y/filter-pruning-geometric-median

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Dynamic Model Pruning with Feedback (2020, methodology): “experimental setup as mentioned in Section 5 with ‘ 2 norm based ﬁlter selection criteria for structured pruning extension on CIFAR-10. We do believe a better ﬁlter selection scheme (Ye et al., 2018; He et al., 2019; Lym et al., 2019) could further improve the results but we leave this exploration f”
Piggyback GAN: Efficient Lifelong Learning for Image Conditioned Generation (2021, background): “There are many effective strategies for improving the parameter efficiency of trained networks: for example, pruning algorithms learn to remove redundant or less important parameters [12,40], and quantization algorithms learn to represent parameters using fewer bits while maintaining task performanc”
Taking a Closer Look at Domain Shift: Category-Level Adversaries for Semantics Consistent Domain Adaptation (2018, background): “To address this issue, domain adaptation approaches [35, 41, 14, 46, 17, 16, 13, 48] are proposed to bridge the gap between the source and target domains.”
Training convolutional neural networks with cheap convolutions and online distillation (2019, methodology): “In addition, OD-Group-2 achieves a comparable result with PFGM, i.e. , 11.33% Top-5 error with 1.12B FLOPs in OD-Group-2 vs. 11.53% Top-5 error with 1.06B FLOPs in PFGM. ResNet-50.”
H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models (2023, positive): “…8-bit integers instead of the commonly employed 32-bit floating point format; ( 2 ) pruning or sparsity (Molchanov et al., 2016; Liu et al., 2018; He et al., 2019; Hoefler et al., 2021), which aims to eliminate unnecessary neurons or weights within the models; ( 3 ) and distillation (Hinton et…”
HRank: Filter Pruning Using High-Rank Feature Map (2020, background): “[12] calculated the geometric median in the layers, and the filters closest to this are pruned.”
Convolution-Weight-Distribution Assumption: Rethinking the Criteria of Channel Pruning (2020, methodology): “, 2019) ||F− Fij ||2 GM (He et al., 2019) ∑Ni+1 k=1 ||Fik − Fij ||2”
Pruning and Quantization for Deep Neural Network Acceleration: A Survey (2021, methodology): “For example, filters that have small norm deviations or a large minimum norm have small search spaces making it difficult to prune based on a threshold [94].”
Greedy Optimization Provably Wins the Lottery: Logarithmic Number of Winning Tickets is Enough (2020, neutral): “(2017b); Li et al. (2017); Molchanov et al. (2019), reconstruction error based method, e.”
Group Fisher Pruning for Practical Network Compression (2021, neutral): “…- FPGM (He et al., 2019) 74.83 76.15 1.90 1.62 IE (Molchanov et al., 2019) 74.50 76.18 2.25 - C-SGD (Ding et al., 2019a) 74.54 75.33 2.20 - Meta (Liu et al., 2019) 75.40 76.60 2.0 - GBN (You et al., 2019) 75.18 75.85 1.84 - LFPC (He et al., 2020) 74.46 76.15 1.60 - HRank (Lin et al., 2020) 74…”
Sim-Real Joint Reinforcement Transfer for 3D Indoor Navigation (2019, methodology): “another kind of early method used in [43, 9, 17], adapt the classifier like SVM trained on the source domain.”
A Survey on Deep Neural Network Pruning: Taxonomy, Comparison, Analysis, and Recommendations (2023, positive): “He et al. [100] propose Filter Pruning via Geometric Median ( FPGM ) to prune redundant ﬁlters that are nearest to the geometric median [101] of the ﬁlters within the same layer.”
Channel Selection Using Gumbel Softmax (2020, methodology): “34 Geom [15] (mix 30%) 73.”
Good Subnetworks Provably Exist: Pruning via Greedy Forward Selection (2020, neutral): “See e.g., Zhuang et al. (2018); Luo et al. (2017); Liu et al. (2017; 2019b); He et al. (2019; 2018b) for examples of recent works.”
Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time (2023, neutral): “…quantization (Han et al., 2015; Jacob et al., 2018; Nagel et al., 2019; Zhao et al., 2019), pruning (Molchanov et al., 2016; Liu et al., 2018; He et al., 2019; Hoefler et al., 2021), and distillation (Hinton et al., 2015; Cho & Hariharan, 2019; Tang et al., 2019; Touvron et al., 2021) have…”
Spatially Adaptive Inference with Stochastic Feature Sampling and Interpolation (2020, methodology): “Compared with SFP [13] and FPGM [14], our approach obtains a smaller accuracy drop with similar FLOPs.”
PP-OCR: A Practical Ultra Lightweight OCR System (2020, methodology): “The implementation of the above FPGM Pruner and PACT quantization is based on PaddleSlim 1 .”