Structured Pruning of Neural Networks With Budget-Aware Regularization

By Carl Lemaire, Andrew Achkar, Pierre-Marc Jodoin. 2018

99 citations recorded by the source corpus.

Pruning methods have shown to be effective at reducing the size of deep neural networks while keeping accuracy almost intact. Among the most effective methods are those that prune a network while training it with a sparsity prior loss and learnable dropout parameters. A shortcoming of these approaches however is that neither the size nor the inference speed of the pruned network can be controlled directly; yet this is a key feature for targeting deployment of CNNs on low-power hardware. To overcome this, we introduce a budgeted regularized pruning framework for deep CNNs. Our approach naturally fits into traditional neural network training as it consists of a learnable masking layer, a novel budget-aware objective function, and the use of knowledge distillation. We also provide insights on how to prune a residual network and how this can lead to new architectures. Experimental results reveal that CNNs pruned with our method are more accurate and less compute-hungry than state-of-the-art methods. Also, our approach is more effective at preventing accuracy collapse in case of severe pruning; this allows pruning factors of up to 16x without significant accuracy drop.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

Controlled Sparsity via Constrained Optimization or: How I Learned to Stop Tuning Penalties and Love Constraints (2022, methodology): “On the other hand, in-training methods which learn a good sparsity pattern by augmenting the training loss with sparsity-inducing penalties [31, 27] do not perform fine-tuning, but face challenges regarding the tuning and interpretability of the penalty hyperparameter.”
Structured Pruning for Deep Convolutional Neural Networks: A Survey (2023, methodology): “To directly control the model budget, Budget-Aware Regular-ization (BAR) [86] uses a prior loss and introduces a learnable dropout parameter θ [236].”
NAT: Neural Architecture Transformer for Accurate and Compact Architectures (2019, methodology): “Several methods [51, 26, 7, 6] have been proposed to obtain compact architectures by learning the optimal settings of each convolution, including kernel size, stride and the number of ﬁlters.”