Dynamic Model Pruning with Feedback

By Tao Lin, Sebastian U. Stich, Luis Barba, D. Dmitriev, Martin Jaggi. 2020

232 citations recorded by the source corpus.

Deep neural networks often have millions of parameters. This can hinder their deployment to low-end devices, not only due to high memory requirements but also because of increased latency at inference. We propose a novel model compression method that generates a sparse trained model without additional overhead: by allowing (i) dynamic allocation of the sparsity pattern and (ii) incorporating feedback signal to reactivate prematurely pruned weights we obtain a performant sparse model in one single training pass (retraining is not needed, but can further improve the performance). We evaluate the method on CIFAR-10 and ImageNet, and show that the obtained sparse models can reach the state-of-the-art performance of dense models and further that their performance surpasses all previously proposed pruning schemes (that come without feedback mechanisms).

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

In Efficiency & Compression

Citations in the corpus

How Well Do Sparse ImageNet Models Transfer? (2021, background): “These mechanisms are very diverse, from surrogates of (cid:96) 0 and (cid:96) 1 - regularization [41,67], to variational methods [53], to meth-ods inspired by compressive sensing mechanisms such as Iterative Hard Thresholding (IHT) [32,33,45,57].”
Structured Pruning for Deep Convolutional Neural Networks: A Survey (2023, background): “In addition to maintaining a sparse dynamic mask, Dynamic Pruning with Feedback (DPF) [117] simultaneously maintains a dense model.”
Layer-adaptive Sparsity for the Magnitude-based Pruning (2020, methodology): “Lin et al. (2020) propose a magnitude-based pruning algorithm using a feedback signal, using a heuristic rule of keeping the last fully connected layer dense.”
WoodFisher: Efficient Second-Order Approximation for Neural Network Compression (2020, methodology): “As followed in the literature [24, 27], we prune only the weights in fully-connected and convolutional layers.”
WoodFisher: Efficient second-order approximations for model compression (2020, methodology): “Dynamic pruning methods (Mostafa and Wang, 2019; Dettmers and Zettlemoyer, 2019; Lin et al., 2020) strive to maintain a sparse model throughout training in comparison to first training a dense model and then pruning it.”
A Survey on Deep Neural Network Pruning: Taxonomy, Comparison, Analysis, and Recommendations (2023, unknown): “2) Dynamic Sparse Training Based Methods: A class of the PDT methods ([81], [91], [92], [94], [95], [96], [97]) take randomly initialized sparse network rather than dense network as the input model.”
Model Pruning Enables Efficient Federated Learning on Edge Devices (2019, methodology): “Considering communication time, dynamic pruning with feedback (DPF) [19] maintains a full-sized model, and clients have to upload full-sized gradients to the server (but only download a subset of model parameters) in every round.”
Soft Threshold Weight Reparameterization for Learnable Sparsity (2020, methodology): “Note that DPF is complementary to most of the techniques discussed here.”
Masking as an Efficient Alternative to Finetuning for Pretrained Language Models (2020, methodology): “Similar to the treatment 2 in Bengio et al. (2013); Hubara et al. (2016); Lin et al. (2020), we use ∂ L ( ˆ W l ) ∂ M l bin as a noisy estimator of ∂ L ( ˆ W l ) ∂ M l to update M l , i.e.: where η refers to the step size.”
Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuning (2021, background): “It is very likely to be confused with model pruning (Li et al., 2017; Zhu and Gupta, 2018; Lin et al., 2020), which also detects a subnetwork within the model (but then removes the other parameters).”