Rethinking the Value of Network Pruning

By Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, Trevor Darrell. 2018

1,668 citations recorded by the source corpus.

Network pruning is widely used for reducing the heavy inference cost of deep models in low-resource settings. A typical pruning algorithm is a three-stage pipeline, i.e., training (a large model), pruning and fine-tuning. During pruning, according to a certain criterion, redundant weights are pruned and important weights are kept to best preserve the accuracy. In this work, we make several surprising observations which contradict common beliefs. For all state-of-the-art structured pruning algorithms we examined, fine-tuning a pruned model only gives comparable or worse performance than training that model with randomly initialized weights. For pruning algorithms which assume a predefined target network architecture, one can get rid of the full pipeline and directly train the target network from scratch. Our observations are consistent for multiple network architectures, datasets, and tasks, which imply that: 1) training a large, over-parameterized model is often not necessary to obtain an efficient final model, 2) learned "important" weights of the large model are typically not useful for the small pruned model, 3) the pruned architecture itself, rather than a set of inherited "important" weights, is more crucial to the efficiency in the final model, which suggests that in some cases pruning can be useful as an architecture search paradigm. Our results suggest the need for more careful baseline evaluations in future research on structured pruning methods. We also compare with the "Lottery Ticket Hypothesis" (Frankle & Carbin 2019), and find that with optimal learning rate, the "winning ticket" initialization as used in Frankle & Carbin (2019) does not bring improvement over random initialization.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

The State of Sparsity in Deep Neural Networks (2019, methodology): “Liu et al. (2018) do explore convolutional architectures on the Ima-geNet datasets, but only at two relatively low sparsity levels (30% and 60%).”
Reducing Transformer Depth on Demand with Structured Dropout (2019, methodology): “DistilBERT occasionally performs worse than BERT of the same size trained from scratch, which confirms the findings of Liu et al. (2018) about the performance of pruned models compared to 1The 48 layer model was trained for 300K updates. training small models from scratch.”
What is the State of Neural Network Pruning? (2020, methodology): “Some methods prune all desired weights at once in a single step (Liu et al., 2019).”
Weight Agnostic Neural Networks (2019, neutral): “Network Pruning By removing connections with small weight values from a trained neural network, pruning approaches [35, 39, 41, 58, 62, 66, 67, 69, 75] can produce sparse networks that keep only a small fraction of the connections, while maintaining similar performance on image classiﬁcation tasks…”
Dynamic Model Pruning with Feedback (2020, background): “ and preprocessing scheme from He et al. (2016a); Huang et al. (2016); for further details refer to Appendix A.2. Models. Following the common experimental setting in related work on network pruning (Liu et al., 2019; Gale et al., 2019; Dettmers & Zettlemoyer, 2019; Mostafa & Wang, 2019), ou”
EfficientDet: Scalable and Efficient Object Detection (2019, background): “Details are in Table 2. or compress existing models [21, 22].”
Network Pruning via Transformable Architecture Search (2019, methodology): “Network pruning [26, 30] is an effective technique to compress and accelerate deep neural networks, and thus allows us to develop the powerful networks [14] on hardware devices with limited storage and computation resources.”
Efficient Structured Pruning and Architecture Searching for Group Convolution (2018, methodology): “We follow the spirit in [25] to either finetune the pruned model or train from scratch the topology.”
Training with Quantization Noise for Extreme Model Compression (2020, background): “[23] for a review of different pruning strategies.”
Linear Mode Connectivity and the Lottery Ticket Hypothesis (2019, background): “Beyond trivial sparsities where many weights remain and random subnetworks can train to full accuracy, sparse networks trained in isolation are generally less accurate than the corresponding dense networks (Han et al., 2015; Li et al., 2017; Liu et al., 2019; Frankle & Carbin, 2019).”
Rigging the Lottery: Making All Tickets Winners (2019, background): “Mozer & Smolensky (1989), LeCun et al.”
Superposition of many models into one (2019, background): “More surprisingly, Frankle & Carbin (2018); Liu et al. (2018) show the pruning mask acquired after training can be applied to the same network before training with little to no adverse impact on learning.”
Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterization (2019, methodology): “(Liu et al., 2018) argues that it is the structure alone of the pruned model that matters, i.e, training a model of the same structure, and starting with random weights, can reach the same level of accuracy as the pruned model.”
Importance Estimation for Neural Network Pruning (2019, background): “The second form allows pruning to automatically find a better architecture, as demonstrated in [24].”
The Lottery Ticket Hypothesis for Pre-trained BERT Networks (2020, background): “Prior work in vision has shown little evidence for the lottery ticket hypothesis when using structured pruning [42]; indeed Prasanna et al. ﬁnd that all subnetworks (“good” and “bad”) have “comparable performance.””
Finding Sparse Structures for Domain Specific Neural Machine Translation (2021, background): “Our approach is also inspired by many studies of sparse networks (Frankle and Carbin 2019; Zhu and Gupta 2018; Liu et al. 2019; Masana et al. 2017).”