Masking as an Efficient Alternative to Finetuning for Pretrained Language Models

By Mengjie Zhao, Tao Lin, Martin Jaggi, Hinrich Schütze. 2020

137 citations recorded by the source corpus.

We present an efficient method of utilizing pretrained language models, where we learn selective binary masks for pretrained weights in lieu of modifying them through finetuning. Extensive evaluations of masking BERT and RoBERTa on a series of NLP tasks show that our masking scheme yields performance comparable to finetuning, yet has a much smaller memory footprint when several tasks need to be inferred simultaneously. Through intrinsic evaluations, we show that representations computed by masked language models encode information necessary for solving downstream tasks. Analyzing the loss landscape, we show that masking and finetuning produce models that reside in minima that can be connected by a line segment with nearly constant test accuracy. This confirms that masking can be utilized as an efficient alternative to finetuning.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

In Natural Language Processing

Citations in the corpus

Prefix-Tuning: Optimizing Continuous Prompts for Generation (2021, background): “One line of research considers removing parameters: some model weights are ablated away by training a binary mask over model parameters (Zhao et al., 2020; Radiya-Dixit and Wang, 2020).”
BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models (2021, methodology): “Zhao et al. (2020) describe a masking-based fine-tuning method, and explicitly mention ignoring the bias terms, as handling them “did not observe a positive effect on performance”.”
Low-Complexity Probing via Finding Subnetworks (2021, background): “More recently, Radiya-Dixit and Wang (2020) and Zhao et al. (2020) showed the analogous result in NLP that weight pruning can be used as an alternative to ﬁne-tuning for pre-trained models.”
PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models (2024, unknown): “PEFT encompasses strategies like partial fine-tuning [15, 16, 17, 18, 19, 20, 21, 22], soft prompt fine-tuning [23, 24, 25, 26, 27, 28, 29], non-linear adapter fine-tuning [30, 31, 32, 33, 34, 35, 36, 37, 38, 39], and low rank adapter based fine-tuning [40, 41, 11, 42].”
Parameter-efficient fine-tuning of large-scale pre-trained language models (2023, methodology): “The masking method 45 learns selective masks for PLMs to only update the critical weights for particular tasks.”
MLPruning: A Multilevel Structured Pruning Framework for Transformer-based Models (2021, methodology): “Most of the previous pruning work in NLP has focused on unstructured pruning [36, 7, 50, 31, 57], rather than structured pruning [27, 11].”
Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment (2023, methodology): “…only the bias term in the attention layer, feed-forward layer and layer normalization of the transformer is updated, Pretrained Weight Masking [36], [37], where the pretrained weights are masked using various pruning criterion, and Delta Weight Masking [38], [39], [40], [41], in which delta…”
Learned Token Pruning for Transformers (2021, methodology): “For GLUE tasks [48], we use 6 tasks for evaluation including sentence similarity (QQP [17], MRPC [8], STS-B [3]), sentiment classification (SST-2 [40]), textual entailment (RTE [6]) and natural language inference (MNLI [52], QNLI [34]).”