BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models

By Elad Ben-Zaken, Shauli Ravfogel, Yoav Goldberg. 2021

1,812 citations recorded by the source corpus.

We introduce BitFit, a sparse-finetuning method where only the bias-terms of the model (or a subset of them) are being modified. We show that with small-to-medium training data, applying BitFit on pre-trained BERT models is competitive with (and sometimes better than) fine-tuning the entire model. For larger data, the method is competitive with other sparse fine-tuning methods.Besides their practical utility, these findings are relevant for the question of understanding the commonly-used process of finetuning: they support the hypothesis that finetuning is mainly about exposing knowledge induced by language-modeling training, rather than learning new task-specific linguistic knowledge.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

In Efficiency & Compression

Citations in the corpus

QLoRA: Efficient Finetuning of Quantized LLMs (2023, methodology): “…(PEFT) methods have been proposed such as prompt tuning [49, 33, 34], tuning the embedding layer inputs [1], tuning hidden states (IA 3 ) [37], adding full layers [27], tuning biases [71], learning a mask over weights based on Fisher information [55], and a combination of approaches [23].”
Flamingo: a Visual Language Model for Few-Shot Learning (2022, background): “[141] Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg.”
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers (2021, methodology): “[33] study a similar method for PLMs that fine-tunes only the biases and the final output layer.”
Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning (2022, methodology): “Specifically, we compare with 9 strong PEFT methods: BitFit [47] which updates only the bias parameters; Adapters [23] which introduce task-specific layers after the self-attention and position-wise feed-forward networks; Compacter and Compacter++ [28] which improve upon adapters by using low-rank m”
Input-Tuning: Adapting Unfamiliar Inputs to Frozen Pretrained Models (2022, neutral): “…the inner structure (e.g., self-attention and feed-forward layers) of the PLM can be modiﬁed, includes Preﬁx-Tuning (Li and Liang, 2021), Bit-ﬁt (Ben Zaken et al., 2021), Child-Tuning (Xu et al., 2021), P-Tuning v2 (Liu et al., 2021b), LoRA (Hu et al., 2021), UniﬁedSKG (Xie et al., 2022) and…”
Training Neural Networks with Fixed Sparse Masks (2021, methodology): “To compare to prior work, we reproduce Bit-Fit [6], in which only the bias parameters of the BERT model are updated across training.”
VL-ADAPTER: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks (2021, methodology): “2 To address this problem, recently, several parameter-efﬁcient training methods have been proposed [12, 16, 17, 20, 28, 33, 47, 52].”
What Language Model to Train if You Have One Million GPU Hours? (2022, background): “Our study is focused on zero-shot use and does not consider efﬁcient ﬁne-tuning (Lester et al., 2021; Zaken et al., 2021), which is quite relevant for large language models, and which may lead to different conclusions.”
Towards a Unified View of Parameter-Efficient Transfer Learning (2021, methodology): “…shows that many methods that claimed comparable results to full ﬁne-tuning on the GLUE benchmark with an encoder-only model (Guo et al., 2021; Ben Zaken et al., 2021; Mahabadi et al., 2021), or on relatively simple generation benchmarks such as E2E (Novikova et al., 2017) with an…”
SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer (2021, background): “Other work explores updating only small parts of the model (Zaken et al., 2021) or task-specific modules, such as adapters (Houlsby et al., 2019; Karimi Mahabadi et al., 2021) or low-rank structures (Mahabadi et al., 2021; Hu et al., 2021), while keeping the rest of the model fixed.”
DEMix Layers: Disentangling Domains for Modular Language Modeling (2021, positive): “This study confirms previous findings that only a subset of LM parameters need to be fine-tuned to a target dataset (Zaken et al., 2021).”
Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! (2023, background): “Numerous ParameterEfficient Fine-Tuning (PEFT) approaches have been developed to further balance the quality and efficiency of this process (Hu et al., 2021; Zaken et al., 2021; Lester et al., 2021; Zhang et al., 2023).”
UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuning (2021, methodology): “Specifically, one may either train a subset of the model parameters such as the prediction head (Lee et al., 2019) and bias terms (Ben Zaken et al., 2021), or introduce task-specific parameters to different parts of the PLM such as before multi-head attention (Li and Liang, 2021) or after feedforwar”