An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-Tuning

By Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, Yue Zhang. IEEE Transactions on Audio, Speech, and Language Processing, 2023

725 citations recorded by the source corpus.

Catastrophic forgetting (CF) is a phenomenon that occurs in machine learning when a model forgets previously learned information while acquiring new knowledge for achieving satisfactory performance in downstream tasks. As large language models (LLMs) have demonstrated remarkable performance, it is intriguing to investigate whether CF exists during the continual instruction tuning of LLMs. This study empirically evaluates the forgetting phenomenon in LLMs’ knowledge during continual instruction tuning from the perspectives of domain knowledge, reasoning, and reading comprehension. The experiments reveal that catastrophic forgetting is generally observed in LLMs ranging from 1 b to 7 b parameters. Surprisingly, as the model scale increases, the severity of forgetting intensifies in such a model scale range, which may result from the much more significant initial performance in the larger LLM. The finding is also observed by the experiment of Qwen-2.5-Inst from 3 B to 14 B. Comparing the decoder-only model BLOOMZ with the encoder-decoder model mT0, BLOOMZ exhibits less forgetting and retains more knowledge. Interestingly, we also observe that LLMs can mitigate language biases, such as gender bias, during continual fine-tuning. Furthermore, our findings indicate that general instruction tuning can help alleviate the forgetting phenomenon in LLMs during subsequent fine-tuning.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/838cd69a0b6c9c244a6eebb0f4742c0625132de6

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Ji-rong Wen
2. Wayne Xin Zhao
3. Prateek Mittal
4. Yutao Zhu
5. Xiangyu Qi
6. Tinghao Xie
7. Pin-Yu Chen
8. Ruoxi Jia

Citations in the corpus

• Don't Make Your LLM an Evaluation Benchmark Cheater (2023, background): “We speculate that it is likely to affect the capacities of LLMs on normally tested tasks (those without data leakage), due to “catastrophe forgetting” (Luo et al., 2023; Goodfel-low et al., 2013).”
• Continual Learning of Large Language Models: A Comprehensive Survey (2024, neutral): “Additionally, IT enhances the interaction between humans and LLMs, providing a more natural interface and aligning LLM outputs more closely with human expectations and preferences [145].”
• Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! (2023, background): “This may arise due to catastrophic forgetting (Kirkpatrick et al., 2017; Luo et al., 2023b) of the initial alignment or due to an inherent tension between the helpfulness and harmlessness objectives (Bai et al., 2022a).”