LoRA: Low-Rank Adaptation of Large Language Models
By J. Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Weizhu Chen, Lu Wang. International Conference on Learning Representations, 2021
21,145 citations recorded by the source corpus.
An important paradigm of natural language processing consists of large-scale pre-training on general domain data and adaptation to particular tasks or domains. As we pre-train larger models, full fine-tuning, which retrains all model parameters, becomes less feasible. Using GPT-3 175B as an example -- deploying independent instances of fine-tuned models, each with 175B parameters, is prohibitively expensive. We propose Low-Rank Adaptation, or LoRA, which freezes the pre-trained model weights and injects trainable rank decomposition matrices into each layer of the Transformer architecture, greatly reducing the number of trainable parameters for downstream tasks. Compared to GPT-3 175B fine-tuned with Adam, LoRA can reduce the number of trainable parameters by 10,000 times and the GPU memory requirement by 3 times. LoRA performs on-par or better than fine-tuning in model quality on RoBERTa, DeBERTa, GPT-2, and GPT-3, despite having fewer trainable parameters, a higher training throughput, and, unlike adapters, no additional inference latency. We also provide an empirical investigation into rank-deficiency in language model adaptation, which sheds light on the efficacy of LoRA. We release a package that facilitates the integration of LoRA with PyTorch models and provide our implementations and model checkpoints for RoBERTa, DeBERTa, and GPT-2 at https://github.com/microsoft/LoRA.
Subfields: Artificial Intelligence, Efficiency & Compression
View Paper: https://arxiv.org/abs/2106.09685
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Luke S. Zettlemoyer
- Quoc V. Le
- Percy Liang
- Yoshua Bengio
- Graham Neubig
- Caiming Xiong
In Efficiency & Compression
Citations in the corpus
- The Flan Collection: Designing Data and Methods for Effective Instruction Tuning (2023, background): “…tuning (a.k.a. continuous or soft “prompt tuning”) shows that only updating a small subset of model parameters can reach comparable performance as fully tuning all model parameters (Lester et al., 2021; Vu et al., 2022; Hu et al., 2021; see He et al., 2022 for a detailed analysis).”
- QLoRA: Efficient Finetuning of Quantized LLMs (2023, methodology): “Our method, QL O RA, uses a novel high-precision technique to quantize a pretrained model to 4-bit, then adds a small set of learnable Low-rank Adapter weights [28] Table 1: Elo ratings for a competition between models, averaged for 10,000 random initial orderings.”
- Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning (2022, methodology): “Specifically, we compare with 9 strong PEFT methods: BitFit [47] which updates only the bias parameters; Adapters [23] which introduce task-specific layers after the self-attention and position-wise feed-forward networks; Compacter and Compacter++ [28] which improve upon adapters by using low-rank m”
- Input-Tuning: Adapting Unfamiliar Inputs to Frozen Pretrained Models (2022, neutral): “Invasive meth-ods, which are built on a strong assumption that the inner structure (e.g., self-attention and feed-forward layers) of the PLM can be modified, includes Prefix-Tuning (Li and Liang, 2021), Bit-fit (Ben Zaken et al., 2021), Child-Tuning (Xu et al., 2021), P-Tuning v2 (Liu et al., 2021b), L”
- VL-ADAPTER: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks (2021, methodology): “Besides the adapter-based methods and full fine-tuning, we also consider prompt-tuning [24] and LoRA [17], which are competitive parameter-efficient training approaches.”
- MAGMA - Multimodal Augmentation of Generative Models through Adapter-based Finetuning (2021, methodology): “Our work builds heavily on recent advances in parameter efficient finetuning of LMs [18,20,21,27,30].”
- LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention (2023, methodology): “Various PEFT techniques have been explored, including prompt tuning [35, 30, 39, 38, 50, 72], Low-Rank Adaptation (LoRA) [23, 69, 20], and adapters [22, 48, 37, 9, 55].”
- Visual Prompt Tuning (2022, background): “On the other hand, given the dominance of large-scale pre-trained Transformerbased Language Models (LM) [14,62,5], many approaches [25,24,31] have been proposed to efficiently fine-tune LM for different downstream NLP tasks [72,71].”
- Fast Model Editing at Scale (2021, background): “Even if g ℓ is a linear network with no hidden layers and produces only a rank-1 parameter edit (motivated by the effectiveness of low-rank model edits observed by Hu et al. (2021)), this function would still require d 2 ( d + d ) = 2 d 3 parameters.”
- DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning (2022, background): “Prompts capture task-specific knowledge with much smaller additional parameters, than its competitors, such as Adapter [57, 42] and LoRA [16].”
- Learning to Prompt for Continual Learning (2021, background): “Prompts capture task-specific knowledge with much smaller additional parameters, than its competitors, such as Adapter [43, 58] and LoRA [18].”
- Towards Continual Knowledge Learning of Language Models (2021, methodology): “LoRA (Hu et al., 2021) We only freeze the encoder for the encoder-decoder LM and the entire model for the decoder-only LM.”
- BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of data (2024, background): “The book that the woman who the man who the child spoke to this morning was reading became a topic of conversation among the friends who had all read it.”
- Audiobox: Unified Audio Generation with Natural Language Prompts (2023, background): “With this approach, only the transcript embedding, projection parameters, along with the LoRA adapter parameters are optimized.”
- PaLM 2 Technical Report (2023, background): “, 2019), and that classifiers are more commonly built with fine-tuning methods or other parameter-efficient methods (Hu et al., 2021; Lester et al., 2021), which we do not evaluate here.”
- Towards a Unified View of Parameter-Efficient Transfer Learning (2021, background): “More recently, Hu et al. (2021) learn low-rank matrices to approximate parameter updates.”
- SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer (2021, background): “Other work explores updating only small parts of the model (Zaken et al., 2021) or task-specific modules, such as adapters (Houlsby et al., 2019; Karimi Mahabadi et al., 2021) or low-rank structures (Mahabadi et al., 2021; Hu et al., 2021), while keeping the rest of the model fixed.”
- ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning for Action Recognition (2022, background): “The third category is learning weight approximation [30].”