Group Fisher Pruning for Practical Network Compression

By Liyang Liu, Shilong Zhang, Zhanghui Kuang, Aojun Zhou, Jingliang Xue, Xinjiang Wang, Yimin Chen, Wenming Yang, Qingmin Liao, Wayne Zhang. 2021

218 citations recorded by the source corpus.

Network compression has been widely studied since it is able to reduce the memory and computation cost during inference. However, previous methods seldom deal with complicated structures like residual connections, group/depth-wise convolution and feature pyramid network, where channels of multiple layers are coupled and need to be pruned simultaneously. In this paper, we present a general channel pruning approach that can be applied to various complicated structures. Particularly, we propose a layer grouping algorithm to find coupled channels automatically. Then we derive a unified metric based on Fisher information to evaluate the importance of a single channel and coupled channels. Moreover, we find that inference speedup on GPUs is more correlated with the reduction of memory rather than FLOPs, and thus we employ the memory reduction of each channel to normalize the importance. Our method can be used to prune any structures including those with coupled channels. We conduct extensive experiments on various backbones, including the classic ResNet and ResNeXt, mobile-friendly MobileNetV2, and the NAS-based RegNet, both on image classification and object detection which is under-explored. Experimental results validate that our method can effectively prune sophisticated networks, boosting inference speed without sacrificing accuracy.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

Training Neural Networks with Fixed Sparse Masks (2021, methodology): “Most relevant to our work, Theis et al. [46] propose utilizing the Fisher to prune and decrease the overall number of parameters for gaze prediction, and Liu et al. [30] also use the Fisher to discover groups of parameters to prune from common backbone architectures.”
A Survey on Deep Neural Network Pruning: Taxonomy, Comparison, Analysis, and Recommendations (2023, unknown): “The update process is as follows: where b is the baseline reward, γ is a discount factor used to avoid over-prioritizing short-term rewards, w Q are the weights of the network Q following Block-QNN [171], and r i is the reward of the whole trajectory for the i th sample.”
Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning (2022, unknown): “To demonstrate the effectiveness and ﬂexibility of our method, we consider several different standard post-training compression scenarios [31, 19, 18].”
LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning (2022, methodology): “In this paper, to keep the whole pruning process efﬁcient, we either use weights magnitude [33] or Fisher Information [51, 35] as importance measures, and reach the target sparsity in one shot.”
SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot (2023, background): “Crucially, following [11], the updated inverse (HUj+1) −1 can be calculated efficiently by removing the first row and column, corresponding to j in the original H, from the inverse of (HUj ) −1 in (3)For example, structured (column-wise) pruning ResNet50 to 50% structured sparsity without accuracy l”