Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers

By Jianbo Ye, Xin Lu, Zhe L. Lin, James Zijun Wang. 2018

432 citations recorded by the source corpus.

Model pruning has become a useful technique that improves the computational efficiency of deep learning, making it possible to deploy solutions in resource-limited scenarios. A widely-used practice in relevant work assumes that a smaller-norm parameter or feature plays a less informative role at the inference time. In this paper, we propose a channel pruning technique for accelerating the computations of deep convolutional neural networks (CNNs) that does not critically rely on this assumption. Instead, it focuses on direct simplification of the channel-to-channel computation graph of a CNN without the need of performing a computationally difficult and not-always-useful task of making high-dimensional tensors of CNN structured sparse. Our approach takes two stages: first to adopt an end-to- end stochastic training method that eventually forces the outputs of some channels to be constant, and then to prune those constant channels from the original neural network by adjusting the biases of their impacting layers such that the resulting compact model can be quickly fine-tuned. Our approach is mathematically appealing from an optimization perspective and easy to reproduce. We experimented our approach through several image learning benchmarks and demonstrate its interesting aspects and competitive performance.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

Dynamic Model Pruning with Feedback (2020, methodology): “lowed the common experimental setup as mentioned in Section 5 with ‘ 2 norm based ﬁlter selection criteria for structured pruning extension on CIFAR-10. We do believe a better ﬁlter selection scheme (Ye et al., 2018; He et al., 2019; Lym et al., 2019) could further improve the results but we leave t”
AtomNAS: Fine-Grained End-to-End Neural Architecture Search (2019, methodology): “To tackle this problem, structured network pruning methods (He et al., 2017b; Liu et al., 2017; Luo et al., 2017; Ye et al., 2018; Gordon et al., 2018) are proposed to prune structured components in networks, e.g. the entire channel and kernel.”
Network Pruning via Transformable Architecture Search (2019, methodology): “[40] applied a ISTA penalty; and He et al.”
Layer-compensated Pruning for Resource-constrained Convolutional Neural Networks (2018, methodology): “As a result, standard training procedures using stochastic gradient descent optimize both Θ and z jointly (Ye et al., 2018; Liu et al., 2017; Gordon et al., 2018).”
Dreaming to Distill: Data-Free Knowledge Transfer via DeepInversion (2019, methodology): “Many techniques have been proposed to successfully compress neural networks [17, 28, 32, 35, 44, 45, 69, 72].”
Rethinking the Value of Network Pruning (2018, methodology): “oup sparsity is also widely used to smooth the pruning process after training (Wen et al., 2016; Alvarez & Salzmann, 2016; Lebedev & Lempitsky, 2016; Zhou et al., 2016). Liu et al. (2017) and Ye et al. (2018) impose sparsity constraints on channel-wise scaling factors during training, whose ”
Importance Estimation for Neural Network Pruning (2019, methodology): “Due to the popularity of batch-normalization [17] layers in recent networks [10, 15], several approaches have been proposed for ﬁlter pruning based on batch-norm parameters [23, 33].”
Discrimination-aware Channel Pruning for Deep Neural Networks (2018, background): “Recent studies on model compression mainly contain three categories, namely, quantization [31, 53], sparse or low-rank compressions [8, 9], and channel pruning [25, 26, 50, 48].”
Normalization Techniques in Training DNNs: Methodology, Analysis and Application (2020, methodology): “[284] also adopted a similar idea, and developed a new algorithmic approach and rescaling trick to improve the robustness and speed of optimization.”
Network Slimming by Slimmable Networks: Towards One-Shot Architecture Search for Channel Numbers (2019, methodology): “Channel pruning ( a.k.a ., network slimming) methods [1, 30, 39, 40, 41] aim at reducing effective channels of a large neural network to speedup its inference.”
Residual Distillation: Towards Portable Deep Neural Networks without Shortcuts (2020, methodology): “Model pruning [15, 16, 17] produces models smaller in size via removing the redundant weights or channel.”
Pruning neural networks: is it time to nip it in the bud? (2018, neutral): “A series of more coarse-grained approaches have been developed to allow the benefits of pruning to be exposed to general purpose devices (He et al., 2017; Ye et al., 2018; Louizos et al., 2017; Liu et al., 2017), all of which revolve around removing entire channels.”
Group Fisher Pruning for Practical Network Compression (2021, methodology): “These methods rely on the “smaller-norm-less-informative” assumption (Ye et al., 2018) which may not be true especially for structured pruning.”
Discrimination-Aware Network Pruning for Deep Model Compression (2020, background): “Recent studies on model compression mainly contain three categories: quantization [19], [59], [87], sparse or low-rank compression [17], [19], [86], and network pruning [46], [51], [81], [83].”
Low-Rank Compression of Neural Nets: Learning the Rank of Each Layer (2020, result): “Importantly, it also puts low-rank compression in perspective with other reported results for faster inference involving structured filter pruning [19, 33, 50, 51, 53].”
Structured Pruning for Deep Convolutional Neural Networks: A Survey (2023, background): “Rethinking Smaller-Norm-Less-Informative (RSNLI) [77] is developed from the basis of previous methods tending to suffer from the Model Reparameterization problem and the Transform Invariance problem.”
Dual Dynamic Inference: Enabling More Efficient, Adaptive, and Controllable Deep Inference (2019, methodology): “More structured pruning was exploited by enforcing group sparsity, such as the filter or channel pruning [21], [22], [23], [24], [25], [26], [27], [28], [29].”
Spatially Adaptive Inference with Stochastic Feature Sampling and Interpolation (2020, background): “Alternatively, sparsity learning techniques identify what to prune in conjunction with network training, through constraints that zero out some filters [36], cause some filters to become redundant and removable [6], scale some filter or block outputs to zero [23], or sparsify batch normalization sca”