Discrimination-aware Channel Pruning for Deep Neural Networks

By Zhuangwei Zhuang, Mingkui Tan, Bohan Zhuang, Jing Liu, Yong Guo, Qingyao Wu, Junzhou Huang, Jin-Hui Zhu. 2018

637 citations recorded by the source corpus.

Channel pruning is one of the predominant approaches for deep model compression. Existing pruning methods either train from scratch with sparsity constraints on channels, or minimize the reconstruction error between the pre-trained feature maps and the compressed ones. Both strategies suffer from some limitations: the former kind is computationally expensive and difficult to converge, whilst the latter kind optimizes the reconstruction error but ignores the discriminative power of channels. To overcome these drawbacks, we investigate a simple-yet-effective method, called discrimination-aware channel pruning, to choose those channels that really contribute to discriminative power. To this end, we introduce additional losses into the network to increase the discriminative power of intermediate layers and then select the most discriminative channels for each layer by considering the additional loss and the reconstruction error. Last, we propose a greedy algorithm to conduct channel selection and parameter optimization in an iterative way. Extensive experiments demonstrate the effectiveness of our method. For example, on ILSVRC-12, our pruned ResNet-50 with 30% reduction of channels even outperforms the original model by 0.39% in top-1 accuracy.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

Data-Efficient Structured Pruning via Submodular Optimization (2022, methodology): “Existing methods either use a heuristic greedy algorithm to solve it [Luo et al., 2017, Zhuang et al., 2018], or they solve instead its (cid:96) 1 -relaxation using alternating minimization [He et al., 2017], or a greedy algorithm with Frank-Wolfe like updates [Ye et al., 2020b].”
DISCO: Dynamic and Invariant Sensitive Channel Obfuscation for deep neural networks (2020, methodology): “While most methods interleave pruning with the training phase [42, 43, 44], there has been recent focus on pruning at inference [45].”
Hierarchical Neural Architecture Search for Single Image Super-Resolution (2020, background): “to real-world applications [18], [19] when the computation resources are limited.”
Continual Learning with Node-Importance based Adaptive Group Sparse Regularization (2020, background): “They have a natural connection with a separate line of research, the model compression of neural networks [17, 19, 42].”
Graph Convolutional Networks for Temporal Action Localization (2019, background): “Recently, great progress has been achieved in deep learning [5, 9, 19, 53], which facilitates the development of temporal action localization.”
Structured Pruning for Deep Convolutional Neural Networks: A Survey (2023, background): “Discrimination-aware Channel Pruning (DCP) [71] aims to keep discriminative channels that substantially change the ﬁnal loss in their absence.”
Attend and Imagine: Multi-Label Image Classification With Visual Attention and Recurrent Neural Networks (2019, background): “In particular, attention mechanism can be viewed as a feature selection strategy [43], [44] which can be used to foucs on the useful information and ignore the redundancies.”
Greedy Optimization Provably Wins the Lottery: Logarithmic Number of Winning Tickets is Enough (2020, neutral): “6 - 175M DPL (Zhuang et al., 2018) 68.”
Channel Gating Neural Networks (2018, methodology): “Many recent proposals suggest pruning unimportant filters/features statically [12, 19, 22, 31].”
Dynamic Channel Pruning: Feature Boosting and Suppression (2018, methodology): “39× Discrimination-aware Channel Pruning (Zhuang et al., 2018) 30.”
DeepHoyer: Learning Sparser Neural Network with Differentiable Scale-Invariant Sparsity Measures (2019, neutral): “05 × DCP (Zhuang et al., 2018) 74.95% 92.32% 2 .”
Cascaded Projection: End-To-End Network Compression and Acceleration (2019, methodology): “Channel pruning methods [35, 37, 39, 44, 56] remove entire feature kernels for network compression.”
Pay Attention to Features, Transfer Learn Faster CNNs (2020, background): “Moreover, by determining the importance of neurons, unimportant ones can further be removed from computation during inference with network pruning methods (Luo et al., 2017; He et al., 2017; Zhuang et al., 2018; Ye et al., 2018; Gao et al., 2019).”
A Survey on Deep Neural Network Pruning: Taxonomy, Comparison, Analysis, and Recommendations (2023, positive): “Discrimination-aware Channel Pruning (DCP) [121] selects the most discriminative channels by minimizing a joint loss which includes the regular loss with the discrimination-aware loss.”
Towards Optimal Structured CNN Pruning via Generative Adversarial Learning (2019, methodology): “To address these issues, techniques have been proposed for CNN compression such as low-rank decomposition [3, 58, 33, 32], parameter quantization [42, 23, 59], knowledge distillation [17, 45] and network pruning [10, 31, 35, 60, 21, 15, 34].”
A Low Effort Approach to Structured CNN Design Using PCA (2018, methodology): “Authors of [33] also remove ﬁlters by introducing multiple losses for each layer that select the most discriminative ﬁlters, but their method iterates until a stopping condition is reached within a layer and iterates over each layer, thus keeping both loops active.”
GAN Compression: Efficient Architectures for Interactive Conditional GANs (2020, methodology): “To further improve the compression ratio, we select the channel width in the generators using automated channel pruning [20, 39, 70, 42] to remove the redundancy, which can reduce the computation quadratically.”
Low-Rank Compression of Neural Nets: Learning the Rank of Each Layer (2020, result): “Importantly, it also puts low-rank compression in perspective with other reported results for faster inference involving structured filter pruning [19, 33, 50, 51, 53].”
Gate Decorator: Global Filter Pruning Method for Accelerating Deep Convolutional Neural Networks (2019, background): “[56] get good results by applies additional discrimination-aware losses to fine-tune the pre-trained model and keep the filters that contribute to the discriminative power.”