Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning

By Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, Tuo Zhao. 2023

729 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Peng Gao
2. Hongsheng Li
3. Yu Qiao
4. Renrui Zhang
5. Ji-rong Wen
6. Wayne Xin Zhao
7. Qun Liu
8. Xin Jiang

In Computer Vision and Pattern Recognition

1. Hongsheng Li
2. Peng Gao
3. Yu Qiao
4. Renrui Zhang
5. Xiaogang Wang
6. Pan Lu
7. Qun Liu
8. Jiaming Han

In Efficiency & Compression

1. Qun Liu
2. Lifeng Shang
3. Xin Jiang
4. Jian-Yun Nie
5. Yu Qiao
6. Peng Gao
7. Ji-rong Wen
8. Liangyou Li

Citations in the corpus

• LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention (2023, methodology): “Various PEFT techniques have been explored, including prompt tuning [35, 30, 39, 38, 50, 72], Low-Rank Adaptation (LoRA) [23, 69, 20], and adapters [22, 48, 37, 9, 55].”
• VeRA: Vector-based Random Matrix Adaptation (2023, result): “Furthermore, similar to previous efficient adaptation methods’ findings Zhang et al. (2023); Liu et al. (2021), we also observe a higher adaptation for the later layers compared to earlier ones.”
• SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuning (2024, neutral): “For instance, tuners incorporate techniques such as LoRA, AdaLoRA [60], IA3 [37], BOFT [39], and Vera [30].”
• ASPEN: High-Throughput LoRA Fine-Tuning of Large Language Models with a Single GPU (2023, positive): “In contrast, parameter-efficient fine-tuning meth-ods like LoRA [19], P-Turning [29], and LoRA’s derivatives including QLoRA [9], SoRA [11], LoftQ [27], and AdaLoRA [52], among others, decrease resource demands while preserving performance.”
• Personalize Segment Anything Model with One Shot (2023, background): “Low-Rank Adaption (LoRA) (Hu et al., 2021; Cuenca & Paul, 2023; Zhang et al., 2023b; Hedegaard et al., 2022) injects trainable rank decomposition matrices concurrently to each pre-trained weight, which significantly reduces the number of learnable parameters required for downstream tasks.”
• Pre-Trained Language Models for Text Generation: A Survey (2022, background): “This flexibility allows for effective adaptation to various down-stream text generation tasks [3, 271].”
• Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment (2023, methodology): “2) LoRA Derivatives: LoRA derivatives refer to a series of PEFT methods that are improved based on LoRA, including Low-Rank Adjustment [44], [45], [46], where different methods are developed to adjust the rank of LoRA dynamically, LoRA-guided Pretrained Weight Update [47], [48], in which LoRA is…”
• PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models (2024, neutral): “PEFT encompasses strategies like partial fine-tuning [15, 16, 17, 18, 19, 20, 21, 22], soft prompt fine-tuning [23, 24, 25, 26, 27, 28, 29], non-linear adapter fine-tuning [30, 31, 32, 33, 34, 35, 36, 37, 38, 39], and low rank adapter based fine-tuning [40, 41, 11, 42].”
• Aligning Large Language Models with Human: A Survey (2023, methodology): “BayLing (Zhang et al., 2023c) introduces a set of multi-turn interactive translation instructions to simultaneously improve multilingual and instruction-following ability for LLMs. Specifically, each multi-turn instruction is essentially a translation task where users first ask LLMs to translate a…”
• Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey (2024, methodology): “Several studies [171], [217], [39], [16], [222], [231] have started to address this issue, but there’s a need for more simple and efficient solutions optimizing these hyperparameters.”