Comparing Rewinding and Fine-tuning in Neural Network Pruning

By Alex Renda, Jonathan Frankle, Michael Carbin. 2020

445 citations recorded by the source corpus.

Many neural network pruning algorithms proceed in three steps: train the network to completion, remove unwanted structure to compress the network, and retrain the remaining structure to recover lost accuracy. The standard retraining technique, fine-tuning, trains the unpruned weights from their final trained values using a small fixed learning rate. In this paper, we compare fine-tuning to alternative retraining techniques. Weight rewinding (as proposed by Frankle et al., (2019)), rewinds unpruned weights to their values from earlier in training and retrains them from there using the original training schedule. Learning rate rewinding (which we propose) trains the unpruned weights from their final values using the same learning rate schedule as weight rewinding. Both rewinding techniques outperform fine-tuning, forming the basis of a network-agnostic pruning algorithm that matches the accuracy and compression ratios of several more network-specific state-of-the-art techniques.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

On the Predictability of Pruning Across Scales (2020, methodology): “, 2015) that gets stateof-the-art tradeoffs between error and unstructured density (Gale et al., 2019; Renda et al., 2020).”
Initialization and Regularization of Factorized Neural Layers (2021, background): “Best of (Frankle & Carbin, 2019; Renda et al., 2020; Su et al., 2020), obtained from Wang et al. (2020) and Su et al. (2020).”
The Lottery Ticket Hypothesis for Pre-trained BERT Networks (2020, methodology): “Renda et al. [18] show that, in other settings, IMP subnetworks rewound early in training reach the same accuracies at the same sparsities as subnetworks found by this standard pruning procedure.”
DepGraph: Towards Any Structural Pruning (2023, background): “In the literature, The design space of pruning algorithms encompasses a range of aspects, including pruning schemes [21, 39], parameter selection [20, 43, 44], layer sparsity [27, 49] and training techniques [47, 58].”
Drawing early-bird tickets: Towards more efficient training of deep networks (2019, unknown): “The comparisons in Table 2 further show that inheriting weights from the EB tickets favor the generalization of retraining as compared to both the random initialization and “over-cooked” weights, aligning well with the recent discussion between rewinding and fine-tuning (Renda et al., 2020).”
Effective Sparsification of Neural Networks with Global Sparsity Constraint (2021, unknown): “[30] achieves strong results but needs multiple rounds of pruning and retraining.”
Gradient Flow in Sparse Neural Networks and How Lottery Tickets Win (2020, background): “This is partially confirmed by Renda, Frankle, and Carbin (2020) which shows that restarting the learning rate schedule from the pruning solution performs better than rewinding the weights.”
Learning N: M Fine-grained Structured Sparse Neural Networks From Scratch (2021, methodology): “Speciﬁcally, we employ the magnitude-based pruning method (Renda et al., 2020; Gale et al., 2019) during the forward process.”
Long Live the Lottery: The Existence of Winning Tickets in Lifelong Learning (2021, methodology): “Moreover, the introduction of weight rewinding is spurred by the so-called rewinding ticket (Renda et al., 2020; Frankle et al., 2020).”
Structured Pruning for Deep Convolutional Neural Networks: A Survey (2023, background): “[272] propose to rewind the learning rate schedule but not the weight value (learning rate rewinding).”
A Simple and Effective Pruning Approach for Large Language Models (2023, methodology): “…either modifications to the training procedure (Sanh et al., 2020; Kusupati et al., 2020), retraining the pruned networks to regain accuracy (Liu et al., 2019; Zhou et al., 2023), or an even more computationally intensive iterative retraining process (Renda et al., 2020; Frankle et al., 2020).”
Sanity-Checking Pruning Methods: Random Tickets can Win the Jackpot (2020, methodology): “And as pointed out in [37], learning rate rewinding usually surpasses weights rewinding, so we mostly focus on learning rate rewinding.”
DiPair: Fast and Accurate Distillation for Trillion-ScaleText Matching and Pair Modeling (2020, background): “Another line of research loosely connected to our work is to reduce inference time via pruning less significant weights and/or converting the model to low-precision (aka quantization) (Han et al., 2016; Howard et al., 2017; Iandola et al., 2016; Renda et al., 2020; Frankle and Carbin, 2019).”