DyLoRA: Parameter-Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptation

By Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, A. Ghodsi. Conference of the European Chapter of the Association for Computational Linguistics, 2022

309 citations recorded by the source corpus.

With the ever-growing size of pretrained models (PMs), fine-tuning them has become more expensive and resource-hungry. As a remedy, low-rank adapters (LoRA) keep the main pretrained weights of the model frozen and just introduce some learnable truncated SVD modules (so-called LoRA blocks) to the model. While LoRA blocks are parameter-efficient, they suffer from two major problems: first, the size of these blocks is fixed and cannot be modified after training (for example, if we need to change the rank of LoRA blocks, then we need to re-train them from scratch); second, optimizing their rank requires an exhaustive search and effort. In this work, we introduce a dynamic low-rank adaptation (DyLoRA) technique to address these two problems together. Our DyLoRA method trains LoRA blocks for a range of ranks instead of a single rank by sorting the representation learned by the adapter module at different ranks during training. We evaluate our solution on different natural language understanding (GLUE benchmark) and language generation tasks (E2E, DART and WebNLG) using different pretrained models such as RoBERTa and GPT with different sizes. Our results show that we can train dynamic search-free models with DyLoRA at least 4 to 7 times (depending to the task) faster than LoRA without significantly compromising performance. Moreover, our models can perform consistently well on a much larger range of ranks compared to LoRA.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/85e959eef45114974c8f8643e88af23936fff3d1

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xiaohui Tao
2. Lingling Xu
3. Jeff Zhang
4. Haoran Xie
5. S. Qin
6. Jinyang Liu
7. Sai Qian Zhang
8. Chao Gao

In Computer Vision and Pattern Recognition

1. Xiaohui Tao
2. Lingling Xu
3. Jeff Zhang
4. Haoran Xie
5. S. Qin
6. Jinyang Liu
7. Sai Qian Zhang
8. Chao Gao

Citations in the corpus

• Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment (2023, methodology): “DyLoRA (Dynamic LoRA) [44] is introduced to overcome two limitations of LoRA: (a) LoRA’s rank is fixed and prevents any changes after training (b) determining the optimal rank for LoRA requires exhaustive search and considerable effort.”
• Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey (2024, methodology): “Several studies [171], [217], [39], [16], [222], [231] have started to address this issue, but there’s a need for more simple and efficient solutions optimizing these hyperparameters.”
• PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models (2024, unknown): “2) Can PiSSA also benefit from some improvements to LoRA, such as AdaLoRA [59] and DyLoRA [67] which adaptively adjust the rank?”