Learning Efficient Convolutional Networks through Network Slimming
By Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, Changshui Zhang. 2017
2,811 citations recorded by the source corpus.
The deployment of deep convolutional neural networks (CNNs) in many real world applications is largely hindered by their high computational cost. In this paper, we propose a novel learning scheme for CNNs to simultaneously 1) reduce the model size; 2) decrease the run-time memory footprint; and 3) lower the number of computing operations, without compromising accuracy. This is achieved by enforcing channel-level sparsity in the network in a simple but effective way. Different from many existing approaches, the proposed method directly applies to modern CNN architectures, introduces minimum overhead to the training process, and requires no special software/hardware accelerators for the resulting models. We call our approach network slimming, which takes wide and large networks as input models, but during training insignificant channels are automatically identified and pruned afterwards, yielding thin and compact models with comparable accuracy. We empirically demonstrate the effectiveness of our approach with several state-of-the-art CNN models, including VGGNet, ResNet and DenseNet, on various image classification datasets. For VGGNet, a multi-pass version of network slimming gives a 20× reduction in model size and a 5× reduction in computing operations.
Subfields: Computer Vision and Pattern Recognition, Efficiency & Compression
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Ming-Hsuan Yang
- Yi Yang
- Xiaogang Wang
- Trevor Darrell
- Alan L. Yuille
- Jitendra Malik
In Efficiency & Compression
Citations in the corpus
- The State of Sparsity in Deep Neural Networks (2019, methodology): “The experiments of Liu et al. (2018) encompass taking the final learned weight mask from a magnitude pruning model, randomly re-initializing the weights, and training the model with the normal training procedure (i.”
- Dynamic Neural Networks: A Survey (2021, methodology): “ation techniques for staticmodels [28], [29], [44], which handle ”easy” and ”hard” inputswith identical computation, and fail to reduce intrinsic computational redundancy.”
- ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design (2018, neutral): “Such discrepancy has been noticed in previous works [7,19,24,30].”
- Distilling Task-Specific Knowledge from BERT into Simple Neural Networks (2019, background): “Thus, others explore pruning entire filters (Li et al., 2016; Liu et al., 2017), with some even targeting device-centric metrics, such as floating-point operations (Tang et al., 2018) and latency (Chen et al., 2018).”
- What is the State of Neural Network Pruning? (2020, background): “Some authors do appear to deliberately point out the type of ResNet they use (e.g., (Liu et al., 2017; Dong et al., 2017)).”
- AtomNAS: Fine-Grained End-to-End Neural Architecture Search (2019, methodology): “To tackle this problem, structured network pruning methods (He et al., 2017b; Liu et al., 2017; Luo et al., 2017; Ye et al., 2018; Gordon et al., 2018) are proposed to prune structured components in networks, e.g. the entire channel and kernel.”
- Once for All: Train One Network and Specialize it for Efficient Deployment (2019, methodology): “…technique for efficient deep learning, including network pruning that removes redundant units (Han et al., 2015) or redundant channels (He et al., 2018; Liu et al., 2017), and quantization that reduces the bit width for the weights and activations (Han et al., 2016; Courbariaux et al., 2015; Zhu…”
- Network Pruning via Transformable Architecture Search (2019, background): “Different criteria for the importance of the filters are applied, such as L2-norm of the filter [27], reconstruction error [17], and learnable scaling factor [29].”
- Efficient Structured Pruning and Architecture Searching for Group Convolution (2018, methodology): “We first compare ResNet-164 with network slimming [24], a state-of-the-art channel-wise structured pruning method based on regularisation.”
- You Only Search Once: Single Shot Neural Architecture Search via Direct Sparse Optimization (2018, background): “For example, as highlighted in (Ma et al. (2018)), memory access may be the bottleneck for lightweight network on GPU because the use of separable convolution.”
- Layer-compensated Pruning for Resource-constrained Convolutional Neural Networks (2018, methodology): “…et al., 2018; Luo et al., 2017; He et al., 2017; Molchanov et al., 2017; He et al., 2018b; Yang et al., 2018a; Jiang et al., 2018; He et al., 2018a; Hu et al., 2016; Liu et al., 2017; Li et al., 2017; Lin et al., 2018; Wen et al., 2016) given its fine-grained control capability over the constraints.”
- ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware (2018, background): “, 2015) or channels (Liu et al., 2017).”
- MorphNet: Fast & Simple Resource-Constrained Structure Learning of Deep Networks (2017, neutral): “This technique was also discovered by a recent work [21] and similar ideas appear else-where [15].”
- Continual Learning with Node-Importance based Adaptive Group Sparse Regularization (2020, background): “They have a natural connection with a separate line of research, the model compression of neural networks [17, 19, 42].”
- Dreaming to Distill: Data-Free Knowledge Transfer via DeepInversion (2019, methodology): “Many techniques have been proposed to successfully compress neural networks [17, 28, 32, 35, 44, 45, 69, 72].”
- Uncertainty-guided Continual Learning with Bayesian Neural Networks (2019, methodology): “In this section, we introduce a variant of our method, UCB-P , which is related to recent efforts in weight pruning in the context of reducing inference computation and network compression (Liu et al., 2017; Molchanov et al., 2016).”
- PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning (2017, methodology): “As neural networks have become deeper and larger, a number of works have emerged aiming to reduce the size of trained models, as well as the computation required for inference, either by reducing the numerical precision required for storing the network weights [5, 6, 12, 23], or by pruning unimporta”
- Essentially No Barriers in Neural Network Energy Landscape (2018, background): “It is also the reason why neural networks can be greatly condensed before a substantial increase in loss occurs (Liu et al., 2017).”
- Rethinking the Value of Network Pruning (2018, methodology): “(Liu et al., 2017).”