LoRA+: Efficient Low Rank Adaptation of Large Models

By Soufiane Hayou, Nikhil Ghosh, Bin Yu. International Conference on Machine Learning, 2024

442 citations recorded by the source corpus.

In this paper, we show that Low Rank Adaptation (LoRA) as originally introduced in Hu et al. (2021) leads to suboptimal finetuning of models with large width (embedding dimension). This is due to the fact that adapter matrices A and B in LoRA are updated with the same learning rate. Using scaling arguments for large width networks, we demonstrate that using the same learning rate for A and B does not allow efficient feature learning. We then show that this suboptimality of LoRA can be corrected simply by setting different learning rates for the LoRA adapter matrices A and B with a well-chosen ratio. We call this proposed algorithm LoRA$+$. In our extensive experiments, LoRA$+$ improves performance (1-2 $\%$ improvements) and finetuning speed (up to $\sim$ 2X SpeedUp), at the same computational cost as LoRA.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/241eefc1bb11e693e0fef6977a65a0a822fb8f5e

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Vitaliy Chiley
2. Daniel King
3. M. Paul
4. Jeff Zhang
5. Jinyang Liu
6. Sai Qian Zhang
7. Chao Gao
8. Zeyu Han

Citations in the corpus

• LoRA Learns Less and Forgets Less (2024, methodology): “One group of methods improves training with LoRA by focusing on initialization or scaling (Meng et al., 2024; Hayou et al., 2024; Li et al., 2023b; Kalajdzievski, 2023; Nikdan et al., 2024), sequential training procedures (Xia et al., 2024), or architectural modifications (Shi et al., 2024).”
• Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey (2024, background): “…Dynamic Rank DyLoRA [171], AdaLoRA [217], SoRA [39], CapaBoost [64], AutoLoRA [221] LoRA Improvement Laplace-LoRA [197], PeriodicLoRA [134], LoRA+ [65], LoRAHub [80], MOELoRA [119], MoLORA [206], MoA [45] Hybrid Fine-tuning UniPELT [133], S4 [16], MAM Adapter [67], NOAH [222], AUTOPEFT [231],…”
• LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models (2024, unknown): “LoRA+ (Hayou et al., 2024) is proposed to overcome the sub-optimality of LoRA.”