QLoRA: Efficient Finetuning of Quantized LLMs
By Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, Luke S. Zettlemoyer. Neural Information Processing Systems, 2023
5,083 citations recorded by the source corpus.
We present QLoRA, an efficient finetuning approach that reduces memory usage enough to finetune a 65B parameter model on a single 48GB GPU while preserving full 16-bit finetuning task performance. QLoRA backpropagates gradients through a frozen, 4-bit quantized pretrained language model into Low Rank Adapters~(LoRA). Our best model family, which we name Guanaco, outperforms all previous openly released models on the Vicuna benchmark, reaching 99.3% of the performance level of ChatGPT while only requiring 24 hours of finetuning on a single GPU. QLoRA introduces a number of innovations to save memory without sacrificing performance: (a) 4-bit NormalFloat (NF4), a new data type that is information theoretically optimal for normally distributed weights (b) double quantization to reduce the average memory footprint by quantizing the quantization constants, and (c) paged optimziers to manage memory spikes. We use QLoRA to finetune more than 1,000 models, providing a detailed analysis of instruction following and chatbot performance across 8 instruction datasets, multiple model types (LLaMA, T5), and model scales that would be infeasible to run with regular finetuning (e.g. 33B and 65B parameter models). Our results show that QLoRA finetuning on a small high-quality dataset leads to state-of-the-art results, even when using smaller models than the previous SoTA. We provide a detailed analysis of chatbot performance based on both human and GPT-4 evaluations showing that GPT-4 evaluations are a cheap and reasonable alternative to human evaluation. Furthermore, we find that current chatbot benchmarks are not trustworthy to accurately evaluate the performance levels of chatbots. A lemon-picked analysis demonstrates where Guanaco fails compared to ChatGPT. We release all of our models and code, including CUDA kernels for 4-bit training.
Subfields: Computer Vision and Pattern Recognition, Electrical and Electronic Engineering, Hardware and Architecture, Efficiency & Compression
View Paper: https://arxiv.org/abs/2305.14314
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Ming-Hsuan Yang
- Trevor Darrell
- Xiaogang Wang
- Jitendra Malik
- Andrew Zisserman
- Yu Qiao
In Electrical and Electronic Engineering
- Rafael Rafailov
- Karl Pertsch
- Suraj Nair
- Kurt Keutzer
- A. Gholami
- Dan Jurafsky
- Jianwei Yang
- Dacheng Tao
In Hardware and Architecture
- Sehoon Kim
- A. Gholami
- Kurt Keutzer
- Tijmen Blankevoort
- Eric P. Xing
- Dacheng Tao
- Chang Zhou
- Noah A. Smith
In Efficiency & Compression
- Kurt Keutzer
- A. Gholami
- Sehoon Kim
- Tijmen Blankevoort
- Joseph E. Gonzalez
- Ion Stoica
- Qun Liu
- Lifeng Shang
Citations in the corpus
- Judging LLM-as-a-judge with MT-Bench and Chatbot Arena (2023, background): “This approach has been tried in our earlier blog post [7] and other concurrent or follow-up work [5, 25, 12, 11, 43, 15].”
- Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey (2024, positive): “QLoRA [124] proposes several novel techniques, including a 4-bit NormalFloat, a Double Quantization, and a Paged Optimizers, to backprop-agate a 4-bit quantized pretrained language model into LoRA.”
- Small Language Models are the Future of Agentic AI (2025, positive): “PEFT techniques such as LoRA [31] or QLoRA [17] can be leveraged to reduce computational costs and memory requirements associated with fine-tuning, making the process more accessible.”
- A Survey on Large Language Models for Code Generation (2024, neutral): “15 different explored prompting techniques are classified into 5 categories depending on their common characteristics, such as root techniques, refinement-based techniques, decomposition-based techniques, reasoning-based techniques, and priming techniques.”
- Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment (2023, methodology): “…Decomposition [11], [42], [43], in which various low-rank decomposition techniques are used to reparameterize the updated matrix, and LoRA derivatives [44], [45], [46], [47], [48], [49], [50], [51], [52], [53], [54], [55], [56], [57], where a series of PEFT methods are developed based on LoRA.”
- KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization (2024, methodology): “These results demonstrate the benefits of our sensitivity-weighted non-uniform quantization approach, relative to NormalFloat quantization [6], as we achieve consistent accuracy improvements of up to 0.32 perplexity across different models (with particularly pronounced improvements for larger…”
- LoRA Learns Less and Forgets Less (2024, methodology): “We compare LoRA and full finetuning after performing an exhaustive learning rate sweep for each method, which we found to be crucial (Dettmers et al., 2024).”
- Aligning Large Language Models with Human: A Survey (2023, methodology): “QLoRA (Dettmers et al., 2023), Unified Prompt (Chen et al.”
- LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models (2024, positive): “A popular solution is efficient fine-tuning (Houlsby et al., 2019; Hu et al., 2022; Dettmers et al., 2023), which reduces the training cost of LLMs when adapting to various tasks.”
- A Survey on Deep Neural Network Pruning: Taxonomy, Comparison, Analysis, and Recommendations (2023, neutral): “Wewillkeepupdatingthis Wewillkeepupdatingthis repositorytoincludethelatestadvancementsinthefield.”
- An Empirical Study of Scaling Instruct-Tuned Large Multimodal Models (2023, methodology): “For LoRA/QLoRA runs, we use a total batchsize of 64 on 1 A100 node for 33B model and 2 nodes for 65B model.”
- A Comprehensive Overview of Large Language Models (2023, positive): “Following this method, LLMs can be easily and effectively extended as multimodal chatbots [208], [202], [210] and multimodal task solvers [211], [212], [213].”
- OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework (2024, methodology): “OpenRLHF seamlessly integrates with Hugging Face Transformers [29] and supports popular technologies such as Mixture of Experts (MoE) [13], Jamba [17] and QLoRA [4].”
- CRAFT Your Dataset: Task-Specific Synthetic Dataset Generation Through Corpus Retrieval and Augmentation (2024, neutral): “Then, using Llama 3 70B Instruct as a judge, we compared win rates, i.e., which output the model preferred: the gold output in the CRAFT datasets or the output generated by Llama 3 8B Instruct.”
- Universal and Transferable Adversarial Attacks on Aligned Language Models (2023, neutral): “For GCG prompts optimized on Vicuna [Zheng et al., 2023] and Guanaco [Dettmers et al., 2023], we measure ASRs on an assortment of comparably-sized open models, including Pythia-12B [Biderman et al., 2023], Falcon-7B [Penedo et al., 2023], ChatGLM-6B [Du et al., 2022], MPT-7B [Team, 2023],…”