"Learning-Compression" Algorithms for Neural Net Pruning

By M. A. Carreira-Perpiñán, Yerlan Idelbayev. 2018

232 citations recorded by the source corpus.

Pruning a neural net consists of removing weights without degrading its performance. This is an old problem of renewed interest because of the need to compress ever larger nets so they can run in mobile devices. Pruning has been traditionally done by ranking or penalizing weights according to some criterion (such as magnitude), removing low-ranked weights and retraining the remaining ones. We formulate pruning as an optimization problem of finding the weights that minimize the loss while satisfying a pruning cost condition. We give a generic algorithm to solve this which alternates "learning" steps that optimize a regularized, data-dependent loss and "compression" steps that mark weights for pruning in a data-independent way. Magnitude thresholding arises naturally in the compression step, but unlike existing magnitude pruning approaches, our algorithm explores subsets of weights rather than committing irrevocably to a specific subset from the beginning. It is also able to learn automatically the best number of weights to prune in each layer of the net without incurring an exponentially costly model selection. Using a single pruning-level user parameter, we achieve state-of-the-art pruning in LeNet and ResNets of various sizes.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

Controlled Sparsity via Constrained Optimization or: How I Learned to Stop Tuning Penalties and Love Constraints (2022, background): “Carreira-Perpinan and Idelbayev [3] consider a reformulation of the constrained optimization problem using “auxiliary variables”, and assume that the constraints enjoy an efficient proximal operator.”
Dynamic Model Pruning with Feedback (2020, methodology): “One variant of this approach is implemented in the Binary Connect algorithm (BC) (Courbariaux et al., 2015) with prominent results, see also Li et al. (2017) for further insights and discussion.”
Sparse Networks from Scratch: Faster Training without Losing Performance (2019, background): “, 2016), joint loss/pruning-cost optimization (Carreira-Perpinán and Idelbayev, 2018), layer-by-layer pruning (Dong et al.”
Artificial neural networks condensation: A strategy to facilitate adaption of machine learning in medical settings by reducing computational burden (2018, methodology): “Same other simple ways including: “Learning-Compression” Algorithms [2], pruning and splicing [3], Net-Trim [10], and L 0 regularization [4] where the parameters are penalized under some norm to generate a optimization problem; Noise-Out [21] where correlation between activation of neurons are…”
Comparing Rewinding and Fine-tuning in Neural Network Pruning (2020, methodology): “Moreover, our pruning algorithm matches the accuracy of pruning algorithms that require more hyperparameters and/or additional methods, such as reinforcement learning (He et al., 2018; Carreira-Perpi˜n´an & Idelbayev, 2018; Zhu & Gupta, 2018).”
Adversarial Network Compression (2018, background): “Reducing the model size (memory and storage) is also the goal of pruning by removing network connections [5,51,60].”
A Low Effort Approach to Structured CNN Design Using PCA (2018, methodology): “…[20] BinarizedNN [21] XNORNet [22] Iannou [23] SNIP [24] Lottery Ticket [25] Imp-Estimation [26] NetTrim [27] Runtime_pruning [28] Learning-comp [29] Learned_L0 [30] Nucleus Nets [31] Structure-learning [32] Discrimin-aware [33] Structure-binary [34] Our Method many of these methodologies are…”
HRank: Filter Pruning Using High-Rank Feature Map (2020, background): “Typical works either prune the filter weights to obtain sparse weight matrices (weight pruning) [10, 9, 1], or remove entire filters from the network (filter pruning) [18, 19, 20].”
Sanity-Checking Pruning Methods: Random Tickets can Win the Jackpot (2020, methodology): “Another line of pruning methods prune the models during training [4, 10, 25, 41].”
Recursive Least Squares for Training and Pruning Convolutional Neural Networks (2022, background): “First, people think it is important to start with training a large, over-parameterized network [22].”
SpArSe: Sparse Architecture Search for CNNs on Resource-Constrained Microcontrollers (2019, methodology): “Pruning [42, 49, 21, 47] is essential to MCU deployment using SpArSe, as it heavily reduces the model size (4) and working memory (5)/(6) without signiﬁ-cantly impacting classiﬁcation accuracy.”
Pruning and Quantization for Deep Neural Network Acceleration: A Survey (2021, methodology): “A similar approach was proposed with an estimation of weights sets [33].”
Low-Rank Compression of Neural Nets: Learning the Rank of Each Layer (2020, methodology): “This formulation now takes the form of model compression as constrained optimization [4, 5, 6], for which a convenient “learning-compression (LC)” optimization algorithm can be generally applied, and which helps us capitalize on the separability of the cost function C.”