ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression

By Jian-Hao Luo, Jianxin Wu, Weiyao Lin. 2017

1,876 citations recorded by the source corpus.

We propose an efficient and unified framework, namely ThiNet, to simultaneously accelerate and compress CNN models in both training and inference stages. We focus on the filter level pruning, i.e., the whole filter would be discarded if it is less important. Our method does not change the original network structure, thus it can be perfectly supported by any off-the-shelf deep learning libraries. We formally establish filter pruning as an optimization problem, and reveal that we need to prune filters based on statistics information computed from its next layer, not the current layer, which differentiates ThiNet from existing methods. Experimental results demonstrate the effectiveness of this strategy, which has advanced the state-of-the-art. We also show the performance of ThiNet on ILSVRC-12 benchmark. ThiNet achieves 3.31 x FLOPs reduction and 16.63× compression on VGG-16, with only 0.52% top-5 accuracy drop. Similar experiments with ResNet-50 reveal that even for a compact network, ThiNet can also reduce more than half of the parameters and FLOPs, at the cost of roughly 1% top-5 accuracy drop. Moreover, the original VGG-16 model can be further pruned into a very small model with only 5.05MB model size, preserving AlexNet level accuracy but showing much stronger generalization ability.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers (2020, methodology): “…factor that our larger models 7 Since the reduction in memory from storing sparse matrices is highly dependent on the data structure used, we follow past work and report the number of nonzero model parameters (Luo et al., 2017; Li et al., 2017). are also less converged on the pretraining task.”
The State of Sparsity in Deep Neural Networks (2019, neutral): “For these reasons, many techniques focus on removing whole neurons and convolutional ﬁlters, or impose block structure on the sparse weights (Liu et al., 2017; Luo et al., 2017; Gray et al., 2017).”
Data-Efficient Structured Pruning via Submodular Optimization (2022, methodology): “Existing methods either use a heuristic greedy algorithm to solve it [Luo et al., 2017, Zhuang et al., 2018], or they solve instead its (cid:96) 1 -relaxation using alternating minimization [He et al., 2017], or a greedy algorithm with Frank-Wolfe like updates [Ye et al., 2020b].”
What is the State of Neural Network Pruning? (2020, methodology): “Similarly, pruning all layers uniformly tends to perform worse than intelligently allocating parameters to different layers (Gale et al., 2019; Han et al., 2015; Li et al., 2016; Molchanov et al., 2016; Luo et al., 2017) or pruning globally (Lee et al., 2019b; Frankle & Carbin, 2019).”
Weight Agnostic Neural Networks (2019, neutral): “Network Pruning By removing connections with small weight values from a trained neural network, pruning approaches [35, 39, 41, 58, 62, 66, 67, 69, 75] can produce sparse networks that keep only a small fraction of the connections, while maintaining similar performance on image classiﬁcation tasks…”
Model-Reuse Attacks on Deep Learning Systems (2018, methodology): “fication system in §6.3 as a concrete example. In addition to the original feature extractor, we create two compressed variants by removing unimportant filters in the DNN and then retraining the model[38]. We set the compression ratio to be 0.75 and 0.5 (i.e., removing 25% and 50% of the filters) fo”
Similarity-Preserving Knowledge Distillation (2019, methodology): “Weight pruning methods [11, 20, 22, 24, 33, 35, 39] remove unimportant weights from the network, sparsifying the network connectivity structure.”
AutoGAN-Distiller: Searching to Compress Generative Adversarial Networks (2020, neutral): “(Luo et al., 2017) also suggested to prune filters based on the statistics from the next layer to evaluate a filter’s importance.”
AtomNAS: Fine-Grained End-to-End Neural Architecture Search (2019, methodology): “To tackle this problem, structured network pruning methods (He et al., 2017b; Liu et al., 2017; Luo et al., 2017; Ye et al., 2018; Gordon et al., 2018) are proposed to prune structured components in networks, e.g. the entire channel and kernel.”
Efficient Structured Pruning and Architecture Searching for Group Convolution (2018, background): “, L1, L2 norms [27, 5, 20, 26], firstorder [29, 3] or second-order [19, 7, 2] gradients, average percentage of zero [26, 11], singular values [32, 28].”
Layer-compensated Pruning for Resource-constrained Convolutional Neural Networks (2018, methodology): “…(Khoram & Li, 2018) in a resource-constrained setting, structural pruning is often adopted as the solution for the problem (Gordon et al., 2018; Luo et al., 2017; He et al., 2017; Molchanov et al., 2017; He et al., 2018b; Yang et al., 2018a; Jiang et al., 2018; He et al., 2018a; Hu et al.,…”
AMC: AutoML for Model Compression and Acceleration on Mobile Devices (2018, background): “Extensive works [20, 19, 34, 12, 18, 17] have been done on accelerating neural networks by compression.”
Dreaming to Distill: Data-Free Knowledge Transfer via DeepInversion (2019, methodology): “ThiNet-70 [35] 2.”
Overcoming Catastrophic Forgetting by Neuron-level Plasticity Control (2019, methodology): “Although there are other methods in network pruning that claim to have better performance(Yu et al., 2018; Luo and Wu, 2017; Luo et al., 2017), we chose the Taylor criterion due to its computational efficiency.”
Continual Learning via Neural Pruning (2019, methodology): “en the different models being superposed in terms of transfer learning. We will provide quantitative comparisons in our experiments. Sparsiﬁcation. Sparsiﬁcation of neural networks has a long history [11, 13, 18]. While sparsiﬁcation is a crucial tool that we use, it is not in itself a focus of this”
Rethinking the Value of Network Pruning (2018, methodology): “We evaluate three pruning methods with predeﬁned target architectures, Li et al. (2017), Luo et al. (2017), He et al. (2017b) and three which automatically discover target models, Liu et al. (2017), Huang & Wang (2018), Han et al. (2015).”
Training with Quantization Noise for Extreme Model Compression (2020, methodology): “On the other hand, structured pruning methods follow the structure of the weights to reduce both the memory footprint and the inference time of a model [9, 21, 22].”