VeRA: Vector-based Random Matrix Adaptation

By Dawid J. Kopiczko, Tijmen Blankevoort, Yuki Markus Asano. International Conference on Learning Representations, 2023

345 citations recorded by the source corpus.

Low-rank adapation (LoRA) is a popular method that reduces the number of trainable parameters when finetuning large language models, but still faces acute storage challenges when scaling to even larger models or deploying numerous per-user or per-task adapted models. In this work, we present Vector-based Random Matrix Adaptation (VeRA), which significantly reduces the number of trainable parameters compared to LoRA, yet maintains the same performance. It achieves this by using a single pair of low-rank matrices shared across all layers and learning small scaling vectors instead. We demonstrate its effectiveness on the GLUE and E2E benchmarks, image classification tasks, and show its application in instruction-tuning of 7B and 13B language models.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/0d7f24578340aae6df610ed95aaa276b9c3ddcd3

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Vitaliy Chiley
2. Daniel King
3. M. Paul
4. Nikhil Ghosh
5. Bin Yu
6. John P. Cunningham
7. Soufiane Hayou

Citations in the corpus

• LoRA Learns Less and Forgets Less (2024, background): “Other works propose alternative low-rank approximations altogether (Liu et al., 2024; Zhao et al., 2024a; Jiang et al., 2024a; Kopiczko et al., 2023).”
• Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey (2024, neutral): “Intrinsic SAID [75], LoRA [76], Compacter [77], KronA [78], KAdaptation [79], HiWi [65], VeRA [80], DoRA [81] LoRA Derivatives Dynamic Rank DyLoRA [82], AdaLoRA [83], SoRA [84], CapaBoost [85], AutoLoRA [86] LoRA Improvement Laplace-LoRA [87], LoRA Dropout [88], PeriodicLoRA [89], LoRA+ [90],…”
• LoRA+: Efficient Low Rank Adaptation of Large Models (2024, background): “Additional variations of LoRA have been proposed such as VeRA (Kopiczko et al., 2023) which freezes random weight tied adapters and learns vector scalings of the internal adapter activations.”