Parameter-efficient fine-tuning of large-scale pre-trained language models

By Ning Ding, Yujia Qin, Guang Yang, Fu Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen and 10 more. Nature Machine Intelligence, 2023

1,192 citations recorded by the source corpus.

With the prevalence of pre-trained language models (PLMs) and the pre-training–fine-tuning paradigm, it has been continuously shown that larger models tend to yield better performance. However, as PLMs scale up, fine-tuning and storing all the parameters is prohibitively costly and eventually becomes practically infeasible. This necessitates a new branch of research focusing on the parameter-efficient adaptation of PLMs, which optimizes a small portion of the model parameters while keeping the rest fixed, drastically cutting down computation and storage costs. In general, it demonstrates that large-scale models could be effectively stimulated by the optimization of a few parameters. Despite the various designs, here we discuss and analyse the approaches under a more consistent and accessible term ‘delta-tuning’, where ‘delta’ a mathematical notation often used to denote changes, is borrowed to refer to the portion of parameters that are ‘changed’ during training. We formally describe the problem and propose a unified categorization criterion for existing delta-tuning methods to explore their correlations and differences. We also discuss the theoretical principles underlying the effectiveness of delta-tuning and interpret them from the perspectives of optimization and optimal control. Furthermore, we provide a holistic empirical study on over 100 natural language processing tasks and investigate various aspects of delta-tuning. With comprehensive study and analysis, our research demonstrates the theoretical and practical properties of delta-tuning in the adaptation of PLMs. Training a deep neural network can be costly but training time is reduced when a pre-trained network can be adapted to different use cases. Ideally, only a small number of parameters needs to be changed in this process of fine-tuning, which can then be more easily distributed. In this Analysis, different methods of fine-tuning with only a small number of parameters are compared on a large set of natural language processing tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/76b19363b10d7ea783e4a6494eae40d73c8e9628

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Peidong Jia
2. Jiaming Liu
3. Ming Lu
4. Senqiao Yang
5. Wei Xue
6. Shanghang Zhang
7. Yandong Guo

In Computer Vision and Pattern Recognition

1. Peidong Jia
2. Jiaming Liu
3. Ming Lu
4. Senqiao Yang
5. Wei Xue
6. Shanghang Zhang
7. Yandong Guo
8. Luc Van Gool

Citations in the corpus

• A Survey on Model Compression for Large Language Models (2023, neutral): “, 2023], DISCO [Chen et al., 2023], SCOTT [Wang et al.”
• Large Language Model (LLM) for Telecommunications: A Comprehensive Survey on Principles, Key Techniques, and Opportunities (2024, methodology): “Therefore, some techniques may be exploited, e.g., efficient parameter-efficient fine-tuning [88], split edge learning [89], and quantized training [90].”
• ViDA: Homeostatic Visual Domain Adapter for Continual Test Time Adaptation (2023, background): “Adapter-based models demonstrate dominant performance over other methods in certain tasks, sometimes surpassing standard fine-tuning [13].”
• Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey (2024, neutral): “From the algorithmic perspective, [25] benchmarks the performance of several PEFT algorithms across more than 100 NLP tasks and conducts systematic experiments based on criteria such as performance, convergence, efficiency, combin-ability, scalability, and transferability.”
• Small Language Models are the Future of Agentic AI (2025, neutral): “Once a satisfactory amount of data has been collected (10k-100k examples being sufficient for fine-tuning of small models as a rule of thumb [5, 19]), it is necessary to remove any PII, PHI, or any other application-specific sensitive data that could cause a data leak across user accounts once used…”
• Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch (2023, neutral): “In this paper, we astonishingly find that Language Models (LMs), similar to Apocalypse and Super Mario, can enhance their capabilities by absorbing other models without the need for retraining or even GPUs.”
• Evaluating Embedding Generalization: How LLMs, LoRA, and SLERP Shape Representational Geometry (2025, unknown): “[7 , 8, 9, 10, 11] These observations have motivated research on model merging and weight-averaging techniques that aim to reconcile the gains of fine-tuning with the robustness of pre-trained checkpoints.”