Channel Pruning for Accelerating Very Deep Neural Networks
By Yihui He, Xiangyu Zhang. 2017
2,781 citations recorded by the source corpus.
In this paper, we introduce a new channel pruning method to accelerate very deep convolutional neural networks. Given a trained CNN model, we propose an iterative two-step algorithm to effectively prune each layer, by a LASSO regression based channel selection and least square reconstruction. We further generalize this algorithm to multi-layer and multi-branch cases. Our method reduces the accumulated error and enhance the compatibility with various architectures. Our pruned VGG-16 achieves the state-of-the-art results by 5× speed-up along with only 0.3% increase of error. More importantly, our method is able to accelerate modern networks like ResNet, Xception and suffers only 1.4%, 1.0% accuracy loss under 2× speedup respectively, which is significant.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Dacheng Tao
- Mohit Bansal
- Quoc V. Le
- Jianfeng Gao
- Yoshua Bengio
- Trevor Darrell
- Luke S. Zettlemoyer
- Percy Liang
In Computer Vision and Pattern Recognition
- Dacheng Tao
- Luc Van Gool
- Yi Yang
- Ming-Hsuan Yang
- Trevor Darrell
- Xiaogang Wang
- Jitendra Malik
- Alan L. Yuille
In Efficiency & Compression
- Song Han
- Yang He
- Thomas S. Huang
- Jiahui Yu
- Jonathan Frankle
- Yi Yang
- Tijmen Blankevoort
- Michael Carbin
Citations in the corpus
- AutoGrow: Automatic Layer Growing in Deep Convolutional Networks (2019, neutral): “Structure pruning and growing were proposed for different goals, such as efficient inference [7, 8, 11, 13, 14, 18, 19, 24, 25, 27, 38–40], lifelong learning [41] and model adaptation [10, 29].”
- Data-Efficient Structured Pruning via Submodular Optimization (2022, methodology): “Existing methods either use a heuristic greedy algorithm to solve it [Luo et al., 2017, Zhuang et al., 2018], or they solve instead its (cid:96) 1 -relaxation using alternating minimization [He et al., 2017], or a greedy algorithm with Frank-Wolfe like updates [Ye et al., 2020b].”
- The Efficiency Misnomer (2021, methodology): “However, it has been shown that using weight decomposition on GPUs can be slower as CUDNN that is optimized for 3 × 3 (He et al., 2017) convolutions and they prefer single large matrix multiplication instead of several small ones.”
- ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design (2018, background): “However, the recent work [7] finds that the decomposition in [36] is even slower on GPU although it reduces FLOPs by 75%.”
- Patient Knowledge Distillation for BERT Model Compression (2019, background): “Network pruning (Han et al., 2015; He et al., 2017), in which network connections are reduced or sparsified, is one common strategy for model compression.”
- What is the State of Neural Network Pruning? (2020, methodology): “…many methods produce pruned models that outperform re-training from scratch with the same sparsity pattern (Zhang et al., 2015; Yu et al., 2018; Louizos et al., 2017; He et al., 2017; Luo et al., 2017; Frankle & Carbin, 2019) (at least with a large enough amount of pruning (Suau et al., 2018)).”
- AutoGAN-Distiller: Searching to Compress Generative Adversarial Networks (2020, positive): “Structured pruning, e.g. channel pruning (Wen et al., 2016; He et al., 2017), was widely adopted as a hardware-friendly compression means (Gui et al., 2016).”
- AtomNAS: Fine-Grained End-to-End Neural Architecture Search (2019, methodology): “To tackle this problem, structured network pruning methods (He et al., 2017b; Liu et al., 2017; Luo et al., 2017; Ye et al., 2018; Gordon et al., 2018) are proposed to prune structured components in networks, e.g. the entire channel and kernel.”
- Network Pruning via Transformable Architecture Search (2019, methodology): “These algorithms lie in two modalities: unstructured pruning [26, 7, 5, 9] and structure pruning [27, 17, 15, 30].”
- Layer-compensated Pruning for Resource-constrained Convolutional Neural Networks (2018, methodology): “Among various model compression methods, structural pruning has been widely adopted to approach the resource-constrained deep neural network optimization problem (Molchanov et al., 2017; He et al., 2017; Luo et al., 2017; He et al., 2018b; Gordon et al., 2018) since it provides solution that works d”
- AMC: AutoML for Model Compression and Acceleration on Mobile Devices (2018, methodology): “We compare our approach with three empirical policies [31, 22] illustrated in Figure 2: uniform sets compression ratio uniformly, shallow and deep aggressively prune shallow and deep layers respectively.”
- Learning to Prune Filters in Convolutional Neural Networks (2018, methodology): “Recently, [15] combined the low-rank approximation with channel pruning and [32] proposed to use Force Regularization to train neural networks towards low-rank spaces.”
- Fast Lossless Neural Compression with Integer-Only Discrete Flows (2022, background): “…network to convergence, scores the weight parameters (usually based on l 1 /l 2 -norms), prunes out low-scored parameters, and fine-tunes the pruned networks (LeCun et al., 1990; Han et al., 2015; Li et al., 2017; Lebedev & Lempitsky, 2016; Wen et al., 2016; He et al., 2017; Frankle & Carbin, 2019).”
- Single Path One-Shot Neural Architecture Search with Uniform Sampling (2019, background): “Previous works usually deal with the problem by channel pruning [8, 9] or model adaption [30, 6].”
- Rethinking the Value of Network Pruning (2018, methodology): “Pruned models such as “VGG-165x” are defined in [4].”
- SkipNet: Learning Dynamic Routing in Convolutional Networks (2017, background): “work focuses on model compression [5,9,12,20] through the application of weight sparsification, filter pruning, vector quantization, and distillation [13] to transfer knowledge to shallower networks.”
- LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networks (2018, methodology): “Representative techniques include quantization [6, 34, 29, 54, 52, 21, 39, 53, 22, 3, 55, 9, 18], pruning [13, 12, 17, 36], low-rank decomposition [7, 8, 24, 27, 38, 51, 49], hashing [4], and deliberate architecture design [23, 19, 50].”