Movement Pruning: Adaptive Sparsity by Fine-Tuning

By Victor Sanh, Thomas Wolf, Alexander M. Rush. 2020

623 citations recorded by the source corpus.

Magnitude pruning is a widely used strategy for reducing model size in pure supervised learning; however, it is less effective in the transfer learning regime that has become standard for state-of-the-art natural language processing applications. We propose the use of movement pruning, a simple, deterministic first-order weight pruning method that is more adaptive to pretrained model fine-tuning. We give mathematical foundations to the method and compare it to existing zeroth- and first-order pruning methods. Experiments show that when pruning large pretrained language models, movement pruning shows significant improvements in high-sparsity regimes. When combined with distillation, the approach achieves minimal accuracy loss with down to only 3% of the model parameters.

Subfields: Artificial Intelligence, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Efficiency & Compression

Citations in the corpus

Learning Compact Metrics for MT (2021, neutral): “Further work includes generalizing the approach other tasks and experimenting with complementary compression methods such as pruning and quantization (Kim et al., 2021; Sanh et al., 2020), as well as increasing linguistic coverage (Joshi et al.”
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (2022, methodology): “Sparse models have seen success in compressing models for inference (pruning) by sparsifying the weight matrices [22, 36, 37, 53, 73].”
It’s Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners (2020, background): “, 2020; Anderson and GómezRodríguez, 2020), pruning (Han et al., 2015, 2016; Sanh et al., 2020) and quantization (Gong et al.”
Parameter-Efficient Transfer Learning with Diff Pruning (2020, background): “…work on learning to mask parts of deep networks with differentiable relaxations of binary masks for model pruning and parameter sharing (Wang et al., 2019b; Zhao et al., 2020; Sanh et al., 2020; Radiya-Dixit & Wang, 2020; Mallya et al., 2018; Guo et al., 2019; Sun et al., 2020a; Cao et al., 2021).”
Understanding and Overcoming the Challenges of Efficient Transformer Quantization (2021, background): “…(Houlsby et al., 2019; Stickland and Murray, 2019), knowledge distillation (Sanh et al., 2019; Jiao et al., 2020), neural architecture search (Guo et al., 2019; Wang et al., 2020a), pruning (Sanh et al., 2020; Prasanna et al., 2020), and better pre-training (Liu et al., 2019; Clark et al., 2020).”
SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer (2021, methodology): “To improve practical applicability of these models, early work introduces compression techniques (Sanh et al., 2019; Jiao et al., 2020; Fan et al., 2020; Sanh et al., 2020) to obtain lightweight models.”
I-BERT: Integer-only BERT Quantization (2021, background): “These techniques can be broadly categorized into: (1) pruning (Fan et al., 2019; Gordon et al., 2020; Han et al., 2015; LeCun et al., 1990; Li et al., 2016b; Mao et al., 2017; 2020; Michel et al., 2019; Molchanov et al., 2016; Raganato et al., 2020; Sanh et al., 2020; Yang et al., 2017); (2) knowled”
Taxonomy of Risks posed by Language Models (2022, background): “ciency gains during training and inference [116], for example via pruning [164], distillation [93, 189], or fine-tuning [148].”
Monarch: Expressive Structured Matrices for Efficient and Accurate Training (2022, methodology): “Sanh et al. (2020) prunes with deterministic firstorder information, which is more adaptive to pretrained model weights.”
Extreme Compression for Pre-trained Transformers Made Simple and Efficient (2022, background): “Many research efforts have been made to compress these huge transformer models including knowledge distillation [20, 54, 53, 61], pruning [57, 43, 8], and low-rank decomposition [30].”
Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compression (2021, methodology): “Many attempts have been made to address this problem, including quantization (Zafrir et al., 2019; Shen et al., 2020), pruning (Michel et al., 2019; Sanh et al., 2020), knowledge distillation (KD) (Sanh et al., 2019; Sun et al., 2019, 2020; Turc et al., 2019; Jiao et al., 2020; Wang et al., 2020;…”
The Low-Resource Double Bind: An Empirical Study of Pruning for Low-Resource Machine Translation (2021, methodology): “Sanh et al. (2020) propose the use of movement pruning, a simple, deterministic ﬁrst-order weight pruning method that is more adaptive to pre-trained model ﬁne-tuning.”
A Simple and Effective Pruning Approach for Large Language Models (2023, methodology): “We choose these two architectures mainly for two reasons: first, as LLMs are based on Transformers, we would like to test if our observations on LLMs still hold on Transformers for other tasks; second, as we are evaluating on image classification, we are interested in examining how these pruning met”
Big Little Transformer Decoder (2023, methodology): “Well-known approaches include efficient architecture design [25, 31, 34, 57, 64, 72], quantization [8, 29, 51, 71, 76, 78, 79], pruning [11, 13, 32, 33, 39, 46, 61], and neural architecture search [5, 53, 54, 63, 73, 77].”
Towards Automated Circuit Discovery for Mechanistic Interpretability (2023, methodology): “…of gradient information being used more generally includes Michel, Levy, and Neubig (2019) who decide which heads should be pruned by using the absolute value of their gradients, while “movement pruning” (Sanh, Wolf, and Rush, 2020) removes parameters that have high velocity to a low magnitude.”
Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Models (2021, methodology): “Subsequent methods [Li et al., 2016, Lin et al., 2017, Dong et al., 2017, Sanh et al., 2020, Lagunas et al., 2021, Zhu and Gupta, 2017] improve on the quality of the pruned models.”
FastFormers: Highly Efficient Transformer Models for Natural Language Understanding (2020, neutral): “Furthermore, there are various papers published to apply this pruning strategy to Transformer models including: (Yu et al., 2019; Sanh et al., 2020; Gordon et al., 2020).”
EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty (2024, background): “…et al., 2015), quantization (Hubara et al., 2018; Shen et al., 2020; Kim et al., 2021; Zadeh et al., 2020; Zafrir et al., 2019), pruning (Gale et al., 2019; Sanh et al., 2020; Kur-tic et al., 2022; Voita et al., 2019), and innovative network architecture designs (Gu & Dao, 2023; Wu et al., 2020).”
DepGraph: Towards Any Structural Pruning (2023, methodology): “In the literature, The design space of pruning algorithms encompasses a range of aspects, including pruning schemes [21, 39], parameter selection [20, 43, 44], layer sparsity [27, 49] and training techniques [47, 58].”