Layer-adaptive Sparsity for the Magnitude-based Pruning

By Jaeho Lee, Sejun Park, Sangwoo Mo, Sungsoo Ahn, Jinwoo Shin. International Conference on Learning Representations, 2020

354 citations recorded by the source corpus.

Recent discoveries on neural network pruning reveal that, with a carefully chosen layerwise sparsity, a simple magnitude-based pruning achieves state-of-the-art tradeoff between sparsity and performance. However, without a clear consensus on"how to choose,"the layerwise sparsities are mostly selected algorithm-by-algorithm, often resorting to handcrafted heuristics or an extensive hyperparameter search. To fill this gap, we propose a novel importance score for global pruning, coined layer-adaptive magnitude-based pruning (LAMP) score; the score is a rescaled version of weight magnitude that incorporates the model-level $\ell_2$ distortion incurred by pruning, and does not require any hyperparameter tuning or heavy computation. Under various image classification setups, LAMP consistently outperforms popular existing schemes for layerwise sparsity selection. Furthermore, we observe that LAMP continues to outperform baselines even in weight-rewinding setups, while the connectivity-oriented layerwise sparsity (the strongest baseline overall) performs worse than a simple global magnitude-based pruning in this case. Code: https://github.com/jaeho-lee/layer-adaptive-sparsity

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/9227d5897abbf297a34d447e94a802a714b8eab2

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xinchao Wang
2. Mingli Song
3. Wai Lam
4. Lidong Bing
5. Fei Huang
6. Gongfan Fang
7. Yongbin Li
8. Michael B. Mi

In Computer Vision and Pattern Recognition

1. Xinchao Wang
2. Mingli Song
3. Fei Huang
4. Wai Lam
5. Gongfan Fang
6. Yongbin Li
7. Michael B. Mi
8. Xinyin Ma

Citations in the corpus

• Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch (2023, methodology): “TIES-Merging (Ya-dav et al., 2023) tackles the task conflicts in Ilharco et al. (2023) by trimming low-magnitude parameters, resolving sign disagreements, and disjointly merging parameters with consistent signs.”
• A Survey on Deep Neural Network Pruning: Taxonomy, Comparison, Analysis, and Recommendations (2023, neutral): “…error in feature maps and formulate the channel pruning problem as follows: where (cid:2) · F is the Frobenius norm, x i is an N × k h k w matrix from i th channel of input x , w i is an n × k h k w weight matrix from i th channel of w , k h and k w are the kernel height and width, respectively.”
• On the Effectiveness of Parameter-Efficient Fine-Tuning (2022, methodology): “Depending on how the parameter-efficient models choose which parameters to optimize, we categorize them into 1) random approaches (Random and Mixout(Lee et al., 2019)), 2) rule-based approaches (BitFit (Zaken et al., 2021), MagPruning (Han et al., 2015a,b; Lee et al., 2021), Adapter (Houlsby et al., 2”
• DepGraph: Towards Any Structural Pruning (2023, background): “In the literature, The design space of pruning algorithms encompasses a range of aspects, including pruning schemes [21, 39], parameter selection [20, 43, 44], layer sparsity [27, 49] and training techniques [47, 58].”
• Structured Pruning for Deep Convolutional Neural Networks: A Survey (2023, background): “Pruning Rate: Structured pruning can also extend the current weight pruning strategies on researching the layer-wise pruning ratios [144], [193], [194].”