Orthogonal Subspace Learning for Language Model Continual Learning

By Xiao Wang, Tianze Chen, Qiming Ge, Han Xia, Rong Bao, Rui Zheng, Qi Zhang, Tao Gui, Xuanjing Huang. Conference on Empirical Methods in Natural Language Processing, 2023

329 citations recorded by the source corpus.

Benefiting from massive corpora and advanced hardware, large language models (LLMs) exhibit remarkable capabilities in language understanding and generation. However, their performance degrades in scenarios where multiple tasks are encountered sequentially, also known as catastrophic forgetting. In this paper, we propose orthogonal low-rank adaptation (O-LoRA), a simple and efficient approach for continual learning in language models, effectively mitigating catastrophic forgetting while learning new tasks. Specifically, O-LoRA learns tasks in different (low-rank) vector subspaces that are kept orthogonal to each other in order to minimize interference. Our method induces only marginal additional parameter costs and requires no user data storage for replay. Experimental results on continual learning benchmarks show that our method outperforms state-of-the-art methods. Furthermore, compared to previous approaches, our method excels in preserving the generalization ability of LLMs on unseen tasks.

View Paper: https://www.semanticscholar.org/paper/28fde851680a40fbbc5c6a44bd3ac6f5ca4ad284

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

  1. Jeff Zhang
  2. Jinyang Liu
  3. Sai Qian Zhang
  4. Chao Gao
  5. Zeyu Han
  6. Fanxu Meng

Citations in the corpus