HRank: Filter Pruning Using High-Rank Feature Map

By Mingbao Lin, Rongrong Ji, Yan Wang, Yichen Zhang, Baochang Zhang, Yonghong Tian, Ling Shao. Computer Vision and Pattern Recognition, 2020

899 citations recorded by the source corpus.

Neural network pruning offers a promising prospect to facilitate deploying deep neural networks on resource-limited devices. However, existing methods are still challenged by the training inefficiency and labor cost in pruning designs, due to missing theoretical guidance of non-salient network components. In this paper, we propose a novel filter pruning method by exploring the High Rank of feature maps (HRank). Our HRank is inspired by the discovery that the average rank of multiple feature maps generated by a single filter is always the same, regardless of the number of image batches CNNs receive. Based on HRank, we develop a method that is mathematically formulated to prune filters with low-rank feature maps. The principle behind our pruning is that low-rank feature maps contain less information, and thus pruned results can be easily reproduced. Besides, we experimentally show that weights with high-rank feature maps contain more important information, such that even when a portion is not updated, very little damage would be done to the model performance. Without introducing any additional constraints, HRank leads to significant improvements over the state-of-the-arts in terms of FLOPs and parameters reduction, with similar accuracies. For example, with ResNet-110, we achieve a 58.2%-FLOPs reduction by removing 59.2% of the parameters, with only a small loss of $0.14\%$ in top-1 accuracy on CIFAR-10. With Res-50, we achieve a 43.8%-FLOPs reduction by removing 36.7% of the parameters, with only a loss of 1.17% in the top-1 accuracy on ImageNet. The codes can be available at https://github.com/lmbxmu/HRank.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/ad4fb74f6a9f110fc3c9037c379da939e6f1ecad

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xinchao Wang
2. Mingli Song
3. Yunhe Wang
4. Chunjing Xu
5. Kai Han
6. Xinjiang Wang
7. Wei Zhang
8. Wenming Yang

In Computer Vision and Pattern Recognition

1. Yunhe Wang
2. Kai Han
3. Xinchao Wang
4. Chunjing Xu
5. Mingli Song
6. Tong Zhang
7. Wei Zhang
8. Xinjiang Wang

Citations in the corpus

• DepGraph: Towards Any Structural Pruning (2023, background): “Among the many network compression paradigms, pruning has proven itself to be highly effective and practical [7,11,29,30,41,54,55,66].”
• Local Relation Learning for Face Forgery Detection (2021, neutral): “With the tremendous success of deep learning (Chollet 2017; Tan and Le 2019; Lin et al. 2020), some works (Afchar et al. 2018; Nguyen, Yamagishi, and Echizen 2019; R¨ossler et al. 2019) adopt neural networks to automatically extract discriminative features for forgery detection.”
• Structured Pruning for Deep Convolutional Neural Networks: A Survey (2023, methodology): “HRank [60] uses the average rank of the current layer’s activation maps from a small set of input data as the filter importance.”
• Model Rubik's Cube: Twisting Resolution, Depth and Width for TinyNets (2020, methodology): “Pruning [11, 23, 39, 25, 45, 47, 24] is used to reduce the redundant parameters in neural networks that are insensitive to the model performance.”
• Group Fisher Pruning for Practical Network Compression (2021, methodology): “…76.18 2.25 - C-SGD (Ding et al., 2019a) 74.54 75.33 2.20 - Meta (Liu et al., 2019) 75.40 76.60 2.0 - GBN (You et al., 2019) 75.18 75.85 1.84 - LFPC (He et al., 2020) 74.46 76.15 1.60 - HRank (Lin et al., 2020) 74 (Luo et al., 2017) 68.17 75.30 1.17 1.35 GDP (Lin et al., 2018) 70.93 75.13 1.57 -”