LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition

By Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, Min Lin. arXiv.org, 2023

393 citations recorded by the source corpus.

Low-rank adaptations (LoRA) are often employed to fine-tune large language models (LLMs) for new tasks. This paper investigates LoRA composability for cross-task generalization and introduces LoraHub, a simple framework devised for the purposive assembly of LoRA modules trained on diverse given tasks, with the objective of achieving adaptable performance on unseen tasks. With just a few examples from a new task, LoraHub can fluidly combine multiple LoRA modules, eliminating the need for human expertise and assumptions. Notably, the composition requires neither additional model parameters nor gradients. Empirical results on the Big-Bench Hard benchmark suggest that LoraHub, while not surpassing the performance of in-context learning, offers a notable performance-efficiency trade-off in few-shot scenarios by employing a significantly reduced number of tokens per example during inference. Notably, LoraHub establishes a better upper bound compared to in-context learning when paired with different demonstration examples, demonstrating its potential for future development. Our vision is to establish a platform for LoRA modules, empowering users to share their trained LoRA modules. This collaborative approach facilitates the seamless application of LoRA modules to novel tasks, contributing to an adaptive ecosystem. Our code is available at https://github.com/sail-sg/lorahub, and all the pre-trained LoRA modules are released at https://huggingface.co/lorahub.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering

View Paper: https://www.semanticscholar.org/paper/3f459219d75de63b5b7a26a8c6447ec1e79a985c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Xiaohui Tao
2. Lingling Xu
3. Jeff Zhang
4. Haoran Xie
5. S. Qin
6. Jinyang Liu
7. Sai Qian Zhang
8. Chao Gao

Citations in the corpus

• Directly Fine-Tuning Diffusion Models on Differentiable Rewards (2023, neutral): “We discuss reward hacking further in Appendix B.8.”
• Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey (2024, background): “…DyLoRA [171], AdaLoRA [217], SoRA [39], CapaBoost [64], AutoLoRA [221] LoRA Improvement Laplace-LoRA [197], PeriodicLoRA [134], LoRA+ [65], LoRAHub [80], MOELoRA [119], MoLORA [206], MoA [45] Hybrid Fine-tuning UniPELT [133], S4 [16], MAM Adapter [67], NOAH [222], AUTOPEFT [231], LLM-Adapters…”
• Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment (2023, methodology): “…and only update W up . based Improvements [52], [53], [54], in which several novel technique are incorporated into LoRA for improvements, and LoRA-based Multi-task Fine-tuning [55], [56], [57], where multiple LoRA modules are combined for cross-task transfer to fine-tune model on a novel task.”