DoRA: Weight-Decomposed Low-Rank Adaptation

By Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, Min-Hung Chen. International Conference on Machine Learning, 2024

957 citations recorded by the source corpus.

Among the widely used parameter-efficient fine-tuning (PEFT) methods, LoRA and its variants have gained considerable popularity because of avoiding additional inference costs. However, there still often exists an accuracy gap between these methods and full fine-tuning (FT). In this work, we first introduce a novel weight decomposition analysis to investigate the inherent differences between FT and LoRA. Aiming to resemble the learning capacity of FT from the findings, we propose Weight-Decomposed Low-Rank Adaptation (DoRA). DoRA decomposes the pre-trained weight into two components, magnitude and direction, for fine-tuning, specifically employing LoRA for directional updates to efficiently minimize the number of trainable parameters. By employing \ours, we enhance both the learning capacity and training stability of LoRA while avoiding any additional inference overhead. \ours~consistently outperforms LoRA on fine-tuning LLaMA, LLaVA, and VL-BART on various downstream tasks, such as commonsense reasoning, visual instruction tuning, and image/video-text understanding. Code is available at https://github.com/NVlabs/DoRA.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/da053e2a4ba1b244940c8f2cad5dcdf0d730f85f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models (2024, positive): “The PEFT (Mangrulkar et al., 2022) library provides an extremely convenient way to implement the adapter-based methods such as LoRA (Hu et al., 2022), rsLoRA (Kalajdzievski, 2023), DoRA (Liu et al., 2024) and PiSSA (Meng et al., 2024).”
Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey (2024, neutral): “Intrinsic SAID [75], LoRA [76], Compacter [77], KronA [78], KAdaptation [79], HiWi [65], VeRA [80], DoRA [81] LoRA Derivatives Dynamic Rank DyLoRA [82], AdaLoRA [83], SoRA [84], CapaBoost [85], AutoLoRA [86] LoRA Improvement Laplace-LoRA [87], LoRA Dropout [88], PeriodicLoRA [89], LoRA+ [90],…”
Small Language Models are the Future of Agentic AI (2025, positive): “Parameter-efficient (e.g., LoRA [30] and DoRA [40]) and full-parameter finetuning for SLMs require only a few GPU-hours, allowing behaviors to be added, fixed, or specialized overnight rather than over weeks [66].”
PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models (2024, neutral): “DoRA [45] introduces a magnitude component to learn the scale of ∆ W while utilizing the original AB as a direction component of ∆ W .”
What matters when building vision-language models? (2024, positive): “We instruction-tune the base model using DoRA (Liu et al., 2024) (a variant of LoRA).”
LoRA Learns Less and Forgets Less (2024, background): “Other works propose alternative low-rank approximations altogether (Liu et al., 2024; Zhao et al., 2024a; Jiang et al., 2024a; Kopiczko et al., 2023).”