GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection

By Jiawei Zhao, Zhenyu (Allen) Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, Yuandong Tian. International Conference on Machine Learning, 2024

500 citations recorded by the source corpus.

Training Large Language Models (LLMs) presents significant memory challenges, predominantly due to the growing size of weights and optimizer states. Common memory-reduction approaches, such as low-rank adaptation (LoRA), add a trainable low-rank matrix to the frozen pre-trained weight in each layer, reducing trainable parameters and optimizer states. However, such approaches typically underperform training with full-rank weights in both pre-training and fine-tuning stages since they limit the parameter search to a low-rank subspace and alter the training dynamics, and further, may require full-rank warm start. In this work, we propose Gradient Low-Rank Projection (GaLore), a training strategy that allows full-parameter learning but is more memory-efficient than common low-rank adaptation methods such as LoRA. Our approach reduces memory usage by up to 65.5% in optimizer states while maintaining both efficiency and performance for pre-training on LLaMA 1B and 7B architectures with C4 dataset with up to 19.7B tokens, and on fine-tuning RoBERTa on GLUE tasks. Our 8-bit GaLore further reduces optimizer memory by up to 82.5% and total training memory by 63.3%, compared to a BF16 baseline. Notably, we demonstrate, for the first time, the feasibility of pre-training a 7B model on consumer GPUs with 24GB memory (e.g., NVIDIA RTX 4090) without model parallel, checkpointing, or offloading strategies.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/c1fa6255cc9fc3128f74befc7855e255bc7a2c6e

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Vitaliy Chiley
2. Daniel King
3. M. Paul
4. John P. Cunningham

In Computer Vision and Pattern Recognition

1. Vitaliy Chiley
2. M. Paul
3. John P. Cunningham
4. David Lopez-Paz

Citations in the corpus

• LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models (2024, unknown): “GaLore achieves lower PPL on large models while LoRA advantages on smaller ones.”
• LoRA Learns Less and Forgets Less (2024, background): “Other works propose alternative low-rank approximations altogether (Liu et al., 2024; Zhao et al., 2024a; Jiang et al., 2024a; Kopiczko et al., 2023).”
• Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey (2024, unknown): “1 [12], QDyLoRA [150], BitDelta [118] Memory-efficientPEFT Side-Tuning [213], LST [167], Res-Tuning [85], MEFT [112], LoRA-FA [215], HyperTuning [146], PEFT Plug-in [88], MeZO [131], GaLore [228] Fig.”