Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks

By Yang He, Guoliang Kang, Xuanyi Dong, Yanwei Fu, Yi Yang. 2018

1,076 citations recorded by the source corpus.

This paper proposed a Soft Filter Pruning (SFP) method to accelerate the inference procedure of deep Convolutional Neural Networks (CNNs). Specifically, the proposed SFP enables the pruned filters to be updated when training the model after pruning. SFP has two advantages over previous works: (1) Larger model capacity. Updating previously pruned filters provides our approach with larger optimization space than fixing the filters to zero. Therefore, the network trained by our method has a larger model capacity to learn from the training data. (2) Less dependence on the pretrained model. Large capacity enables SFP to train from scratch and prune the model simultaneously. In contrast, previous filter pruning methods should be conducted on the basis of the pre-trained model to guarantee their performance. Empirically, SFP from scratch outperforms the previous filter pruning methods. Moreover, our approach has been demonstrated effective for many advanced CNN architectures. Notably, on ILSCRC-2012, SFP reduces more than 42% FLOPs on ResNet-101 with even 0.2% top-5 accuracy improvement, which has advanced the state-of-the-art. Code is publicly available on GitHub: https://github.com/he-y/softfilter-pruning

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

What is the State of Neural Network Pruning? (2020, methodology): “These values are sometimes wall time, sometimes original number of FLOPs divided by pruned number of FLOPs, sometimes a more complex formula relating these two quantities (Dong et al., 2017; He et al., 2018a), and some- times never made clear.”
Dynamic Model Pruning with Feedback (2020, methodology): “Baselines Model Dense SFP (He et al., 2018) DPF Pruning ratio ResNet-20 92.”
Network Pruning via Transformable Architecture Search (2019, methodology): “, pruning 30% filters in each layer [27, 15], predicting sparsity ratio [12], or leveraging regularization [2].”
Layer-compensated Pruning for Resource-constrained Convolutional Neural Networks (2018, methodology): “We randomly sample 3k images from the training set , which follows (He et al., 2018b), to evaluate the objective in (9) for each candidate.”
FBNetV2: Differentiable Neural Architecture Search for Spatial and Channel Dimensions (2020, methodology): “However, with heuristicsbased simplifications, pruning methods train potential architectures separately, one after another – in some cases, pruning methods consider only one architecture [22, 10].”
Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey (2020, methodology): “Different from the hard pruning, the pruned filters can also be invoked during gradient descent, termed soft filter pruning [212].”
Dynamic Sparse Training: Find Efficient Sparse Network From Scratch With Trainable Masked Layers (2020, background): “He et al. (2018) propose ”soft” pruning algorithm that allows the pruned connection to be updated in later back-propagation, which only solves the first problem.”
Masking as an Efficient Alternative to Finetuning for Pretrained Language Models (2020, background): “Investigating the lottery ticket hypothesis (Fran-kle and Carbin, 2018) of network pruning (Han et al., 2015a; He et al., 2018; Liu et al., 2019d; Lee et al., 2019; Lin et al., 2020), Zhou et al. (2019) ﬁnd that applying binary masks to a neural network is a form of training the network.”
Structured Pruning for Deep Convolutional Neural Networks: A Survey (2023, background): “For example, in 2018, SFP [115] reduced 41.”
Learning Low-rank Deep Neural Networks via Singular Vector Orthogonality Regularization and Singular Value Sparsification (2020, methodology): “Here we mainly compare our method with state-of-the-art low-rank decomposition methods including Jaderberg et al. [15], Zhang et al. [39], TRP [35] and C-SGD [5], as well as recent ﬁlter pruning methods like NISP [37], SFP [10] and CNN-FCF [20].”
CAP: Context-Aware Pruning for Semantic Segmentation (2021, background): “SFP [13] keeps updating the pruned filters by setting their weights to zero instead of removing.”
Cascaded Projection: End-To-End Network Compression and Acceleration (2019, result): “We compare the results obtained by CaP with those of [17, 18, 35, 54, 19] where the networks have been subjected to similar compression ratios.”
Differentiable Joint Pruning and Quantization for Hardware Efficiency (2020, methodology): “In [18] and [11], the pruned channels are determined with L1 and L2 norm, respectively for each layer under the same pruning ratio.”
Gate Decorator: Global Filter Pruning Method for Accelerating Deep Convolutional Neural Networks (2019, neutral): “72 SFP [12] 7 74.”
Spatially Adaptive Inference with Stochastic Feature Sampling and Interpolation (2020, methodology): “Compared with SFP [12] and FPGM [13], our approach obtains a smaller accuracy drop with similar FLOPs.”
Dynamic Slimmable Network (2021, background): “In recent years, many different approaches have been proposed to improve the inference efficiency of neural networks, including network pruning [40, 47, 19, 20, 48, 50], weight quantization [30], knowledge distillation [2, 52, 21], manually and automatically designing of efficient networks [56, 53, ”
CGaP: Continuous Growth and Pruning for Efficient Deep Learning (2019, result): “On excessively deep networks like ResNet-110, CGaP further reduce 63.3% FLOPs compared to 40.8% reported by [He et al., 2018].”
Drawing early-bird tickets: Towards more efficient training of deep networks (2019, positive): “Popular means include pruning (Li et al., 2017; Liu et al., 2017; He et al., 2018; Wen et al., 2016; Luo et al., 2017; Liu et al., 2018a), weight factorization (Denton et al.”
Pruning and Quantization for Deep Neural Network Acceleration: A Survey (2021, methodology): “Soft filter pruning (SFP) [93] further extended recoverable pruning using a dimension of filter.”