Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey
By Zeyu Han, Chao Gao, Jinyang Liu, Jeff Zhang, Sai Qian Zhang. Trans. Mach. Learn. Res., 2024
1,015 citations recorded by the source corpus.
Large models represent a groundbreaking advancement in multiple application fields, enabling remarkable achievements across various tasks. However, their unprecedented scale comes with significant computational costs. These models, often consisting of billions of parameters, require vast amounts of computational resources for execution. Especially, the expansive scale and computational demands pose considerable challenges when customizing them for particular downstream tasks, particularly over the hardware platforms constrained by computational capabilities. Parameter Efficient Fine-Tuning (PEFT) provides a practical solution by efficiently adjusting the large models over the various downstream tasks. In particular, PEFT refers to the process of adjusting the parameters of a pre-trained large model to adapt it to a specific task or domain while minimizing the number of additional parameters introduced or computational resources required. This approach is particularly important when dealing with large-scale language models with high parameter counts, as fine-tuning these models from scratch can be computationally expensive and resource-intensive, posing considerable challenges in the supporting system platform design. In this survey, we present comprehensive studies of various PEFT algorithms, examining their performance and computational overhead. Moreover, we provide an overview of applications developed using different PEFT algorithms and discuss common techniques employed to mitigate computation costs for PEFT. In addition to providing an extensive survey from an algorithmic standpoint, we also examine various real-world system designs to investigate the implementation costs associated with different PEFT approaches. This survey serves as a valuable resource for researchers aiming to understand both the PEFT algorithm and its system implementation, offering detailed ......
Subfields: Artificial Intelligence, Hardware and Architecture, Statistical and Nonlinear Physics
View Paper: https://www.semanticscholar.org/paper/916b4926cda574dc3f9486bb9994b6f2788dd800
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
Citations in the corpus
- A Survey on Mixture of Experts in Large Language Models (2024, neutral): “This challenge has led to the development of parameter-efficient fine-tuning (PEFT) methods, which update only a small set of parameters, achieving remarkable performances in many NLP tasks with less computational cost [150], [151].”
- PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models (2024, positive): “Consequently, various parameter-efficient fine-tuning (PEFT) [12, 13] methods have been proposed to reduce the number of parameters and memory usage required for fine-tuning.”
- Query-efficient model evaluation using cached responses (2026, unknown): “At the same time, it is easier than ever to produce new model behavior – via parameter efficient fine-tuning (Han et al., 2024), instruction tuning (Zhang et al., 2025), model merging (Matena & Raffel, 2022), distillation (Gou et al., 2021), prompt engineering (Sahoo et al., 2025), etc. – making it…”