On the Effectiveness of Parameter-Efficient Fine-Tuning

By Zihao Fu, Haoran Yang, Anthony Man-Cho So, Wai Lam, Lidong Bing, Nigel Collier. 2022

249 citations recorded by the source corpus.

Fine-tuning pre-trained models has been ubiquitously proven to be effective in a wide range of NLP tasks. However, fine-tuning the whole model is parameter inefficient as it always yields an entirely new model for each task. Currently, many research works propose to only fine-tune a small portion of the parameters while keeping most of the parameters shared across different tasks. These methods achieve surprisingly good performance and are shown to be more stable than their corresponding fully fine-tuned counterparts. However, such kind of methods is still not well understood. Some natural questions arise: How does the parameter sparsity lead to promising performance? Why is the model more stable than the fully fine-tuned models? How to choose the tunable parameters? In this paper, we first categorize the existing methods into random approaches, rule-based approaches, and projection-based approaches based on how they choose which parameters to tune. Then, we show that all of the methods are actually sparse fine-tuned models and conduct a novel theoretical analysis of them. We indicate that the sparsity is actually imposing a regularization on the original model by controlling the upper bound of the stability. Such stability leads to better generalization capability which has been empirically observed in a lot of recent research works. Despite the effectiveness of sparsity grounded by our theory, it still remains an open problem of how to choose the tunable parameters. Currently, the random and rule-based methods do not utilize task-specific data information while the projection-based approaches suffer from the projection discontinuity problem. To better choose the tunable parameters, we propose a novel Second-order Approximation Method (SAM) which approximates the original problem with an analytically solvable optimization function. The tunable parameters are determined by directly optimizing the approximation function. We conduct extensive experiments on several tasks. The experimental results show that our proposed SAM model outperforms many strong baseline models and it also verifies our theoretical analysis. The source code of this paper can be obtained from https://github.com/fuzihaofzh/AnalyzeParameterEff\/icientFinetune .

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Ee-Peng Lim
2. Soujanya Poria
3. Yihuai Lan
4. Lei Wang
5. Zhiqiang Hu
6. Sri Karlapati
7. Lidong Bing
8. Xiaohui Tao

In Computer Vision and Pattern Recognition

1. Lei Wang
2. Zhiqiang Hu
3. Sri Karlapati
4. Ee-Peng Lim
5. Soujanya Poria
6. Lidong Bing
7. Xiaohui Tao
8. Alexis Moinet

Citations in the corpus

• BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of data (2024, background): “The book that the woman who the man who the child spoke to this morning was reading became a topic of conversation among the friends who had all read it.”
• Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment (2023, methodology): “SAM (Second-order Approximation Method) [41] also employs the sparse mask matrix to update the delta weight.”
• PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models (2024, unknown): “PEFT encompasses strategies like partial fine-tuning [15, 16, 17, 18, 19, 20, 21, 22], soft prompt fine-tuning [23, 24, 25, 26, 27, 28, 29], non-linear adapter fine-tuning [30, 31, 32, 33, 34, 35, 36, 37, 38, 39], and low rank adapter based fine-tuning [40, 41, 11, 42].”
• LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models (2023, methodology): “To overcome this challenge, Alpaca-LoRA3 proposes integrating a parameter-efficient fine-tuning (PEFT) (Houlsby et al., 2019; Lester et al., 2021; Mangrulkar et al., 2022; Fu et al., 2022) method, Low-Rank Adaptation (LoRA) (Hu et al.”
• Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey (2024, methodology): “SAM [48] proposes a second-order approximation method, which approximates the original problem with an analytically solvable optimization function, to help decide the parameter mask.”
• SemRAG: Semantic Knowledge-Augmented RAG for Improved Question-Answering (2025, unknown): “PEFT relies on the model’s pre-existing knowledge base, which may not suffice for domains requiring constant access to updated or specialized information [8].”