Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment
By Lingling Xu, Haoran Xie, S. Qin, Xiaohui Tao, Fu Lee Wang. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023
370 citations recorded by the source corpus.
With the continuous growth in the number of parameters of the Transformer-based pretrained language models (PLMs), particularly the emergence of large language models (LLMs) with billions of parameters, many natural language processing (NLP) tasks have demonstrated remarkable success. However, the enormous size and computational demands of these models pose significant challenges for adapting them to specific downstream tasks, especially in environments with limited computational resources. Parameter-Efficient Fine-Tuning (PEFT) offers an effective solution by reducing the number of fine-tuning parameters and memory usage while achieving comparable performance to full fine-tuning. The demands for fine-tuning PLMs, especially LLMs, have led to a surge in the development of PEFT methods, as depicted in Fig. 1. In this paper, we present a comprehensive and systematic review of PEFT methods for PLMs. We summarize these PEFT methods, discuss their applications, and outline future directions. Furthermore, extensive experiments are conducted using several representative PEFT methods to better understand their effectiveness in parameter efficiency and memory efficiency. By offering insights into the latest advancements and practical applications, this survey serves as an invaluable resource for researchers and practitioners seeking to navigate the challenges and opportunities presented by PEFT in the context of PLMs.
Subfields: Artificial Intelligence
View Paper: https://www.semanticscholar.org/paper/83de13bd492b9e72c314e308f0d77014154a6a74
Citations in the corpus
- A Survey on the Memory Mechanism of Large Language Model-based Agents (2024, neutral): “In addition, cognitive psychology has been studied for a long time, so many effective human memory theories and architectures have been accumulated, which can support more advanced capabilities of the agents [147].”
- Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey (2024, neutral): “Similarly, [26] and [27] have also established targeted benchmarks to evaluate different PEFT algorithms.”
- PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models (2024, neutral): “Consequently, various parameter-efficient fine-tuning (PEFT) [12, 13] methods have been proposed to reduce the number of parameters and memory usage required for fine-tuning.”