Parameter-Efficient Transfer Learning with Diff Pruning

By Demi Guo, Alexander M. Rush, Yoon Kim. 2020

519 citations recorded by the source corpus.

The large size of pretrained networks makes them difficult to deploy for multiple tasks in storage-constrained settings. Diff pruning enables parameter-efficient transfer learning that scales well with new tasks. The approach learns a task-specific “diff” vector that extends the original pretrained parameters. This diff vector is adaptively pruned during training with a differentiable approximation to the L0-norm penalty to encourage sparsity. As the number of tasks increases, diff pruning remains parameter-efficient, as it requires storing only a small diff vector for each task. Since it does not require access to all tasks during training, it is attractive in on-device deployment settings where tasks arrive in stream or even from different providers. Diff pruning can match the performance of finetuned baselines on the GLUE benchmark while only modifying 0.5% of the pretrained model’s parameters per task and scales favorably in comparison to popular pruning approaches.

Subfields: Artificial Intelligence, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Efficiency & Compression

Citations in the corpus

Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning (2022, methodology): “Since then, various sophisticated PEFT methods have been proposed, including methods that choose a sparse subset of parameters to train [25, 26], produce low-rank updates [13], perform optimization in a lower-dimensional subspace [27], add low-rank adapters using hypercomplex multiplication [28], an”
Training Neural Networks with Fixed Sparse Masks (2021, background): “5% is achieved by Diff Pruning [16].”
VL-ADAPTER: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks (2021, methodology): “2 To address this problem, recently, several parameter-efﬁcient training methods have been proposed [12, 16, 17, 20, 28, 33, 47, 52].”
The Efficiency Misnomer (2021, methodology): “…of Parameters: Number of trainable parameters is also used as an indirect indicator of computational complexity as well as memory usage (during inference) (Kim et al., 2021; Arnab et al., 2021; Tan and Le, 2019; Liu et al., 2021; Guo et al., 2020; Mahabadi et al., 2021a;b; Houlsby et al., 2019).”
Towards a Unified View of Parameter-Efficient Transfer Learning (2021, methodology): “This shows that many methods that claimed comparable results to full fine-tuning on the GLUE benchmark with an encoder-only model (Guo et al., 2021; Ben Zaken et al., 2021; Mahabadi et al., 2021), or on relatively simple generation benchmarks such as E2E (Novikova et al., 2017) with an…”
AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning (2023, methodology): “This results in similar computational cost as full fine-tuning (Guo et al., 2020).”
Block Pruning For Faster Transformers (2021, methodology): “Approaches such as magnitude pruning (Han et al., 2015), L0 regularization (Louizos et al., 2018), lottery ticket hypothesis (Frankle and Carbin, 2018), diff pruning (Guo et al., 2020), and movement pruning (Sanh et al., 2020) have demonstrated remarkable reductions in model size.”
UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuning (2021, methodology): “Other studies take a step further by freezing the entire PLM and adding a small number of additional trainable parameters (Houlsby et al., 2019; Li and Liang, 2021; Lester et al., 2021; Guo et al., 2021; Hu et al., 2021).”
Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning (2023, unknown): “This results in similar computational cost as full fine-tuning (Guo et al., 2020).”
BERT Busters: Outlier Dimensions that Disrupt Transformers (2021, methodology): “After the initial reports of redundancy in the BERT model (Michel et al., 2019; Kovaleva et al., 2019), compressing Trans-formers quickly became a subﬁeld of its own (Jiao et al., 2020; Zafrir et al., 2019; Fan et al., 2019; Guo et al., 2020).”
AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition (2022, background): “The pioneer works could be mainly categorized from the token [58, 56] and network perspectives [46, 47, 92, 40].”
On the Effectiveness of Parameter-Efficient Fine-Tuning (2022, methodology): “It should be noted that as indicated by (Guo et al., 2021; Houlsby et al., 2019), though most of the parameter-eﬃcient models need more time to train, they actually need less storage space as they can just only store the changed parameters.”
PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models (2024, unknown): “PEFT encompasses strategies like partial fine-tuning [15, 16, 17, 18, 19, 20, 21, 22], soft prompt fine-tuning [23, 24, 25, 26, 27, 28, 29], non-linear adapter fine-tuning [30, 31, 32, 33, 34, 35, 36, 37, 38, 39], and low rank adapter based fine-tuning [40, 41, 11, 42].”
LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning (2022, methodology): “Parameter-efﬁcient transfer learning (PETL) [31, 23, 39, 40, 34, 51, 58, 17, 24, 18, 42, 60, 14, 52, 59] is a recent research direction for online or multi-task learning.”
Parameter-efficient fine-tuning of large-scale pre-trained language models (2023, methodology): “Some delta-tuning methods can be categorized into this approach, for example, LoRA 15 , BitFit 14 and diff pruning 44 .”