Neural Pruning via Growing Regularization

By Huan Wang, Can Qin, Yulun Zhang, Yun Raymond Fu. 2020

193 citations recorded by the source corpus.

Regularization has long been utilized to learn sparsity in deep neural network pruning. However, its role is mainly explored in the small penalty strength regime. In this work, we extend its application to a new scenario where the regularization grows large gradually to tackle two central problems of pruning: pruning schedule and weight importance scoring. (1) The former topic is newly brought up in this work, which we find critical to the pruning performance while receives little research attention. Specifically, we propose an L2 regularization variant with rising penalty factors and show it can bring significant accuracy gains compared with its one-shot counterpart, even when the same weights are removed. (2) The growing penalty scheme also brings us an approach to exploit the Hessian information for more accurate pruning without knowing their specific values, thus not bothered by the common Hessian approximation problems. Empirically, the proposed algorithms are easy to implement and scalable to large datasets and networks in both structured and unstructured pruning. Their effectiveness is demonstrated with modern deep neural networks on the CIFAR and ImageNet datasets, achieving competitive results compared to many state-of-the-art algorithms. Our code and trained models are publicly available at this https URL.

Subfields: Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Efficiency & Compression

Citations in the corpus

Controlled Sparsity via Constrained Optimization or: How I Learned to Stop Tuning Penalties and Love Constraints (2022, methodology): “Wang et al. [44] start by identifying the parameters to be removed by applying magnitude pruning on a pre-trained model.”
DepGraph: Towards Any Structural Pruning (2023, methodology): “In the literature, The design space of pruning algorithms encompasses a range of aspects, including pruning schemes [21, 39], parameter selection [20, 43, 44], layer sparsity [27, 49] and training techniques [47, 58].”
Structured Pruning for Deep Convolutional Neural Networks: A Survey (2023, unknown): “Growing Regularization (GREG) [94] exploits regularization under a growing penalty and uses two algorithms.”
A Survey on Deep Neural Network Pruning: Taxonomy, Comparison, Analysis, and Recommendations (2023, unknown): “…learn pruning masks that match the target architecture and maximize performance by jointly optimizing weights and pruning masks with a min-max objective for LLMs. Somemethods([38],[53],[105],[108])selectimportantweights (orﬁlters,neurons,etc.)bycombiningnorm-basedcriteriaand sparsityregularization.”