Continual Pre-Training Mitigates Forgetting in Language and Vision

By Andrea Cossu, Tinne Tuytelaars, Antonio Carta, Lucia C. Passaro, Vincenzo Lomonaco, Davide Bacciu. 2022

111 citations recorded by the source corpus.

Pre-trained models are commonly used in Continual Learning to initialize the model before training on the stream of non-stationary data. However, pre-training is rarely applied during Continual Learning. We investigate the characteristics of the Continual Pre-Training scenario, where a model is continually pre-trained on a stream of incoming data and only later fine-tuned to different downstream tasks. We introduce an evaluation protocol for Continual Pre-Training which monitors forgetting against a Forgetting Control dataset not present in the continual stream. We disentangle the impact on forgetting of 3 main factors: the input modality (NLP, Vision), the architecture type (Transformer, ResNet) and the pre-training protocol (supervised, self-supervised). Moreover, we propose a Sample-Efficient Pre-training method (SEP) that speeds up the pre-training phase. We show that the pre-training protocol is the most important factor accounting for forgetting. Surprisingly, we discovered that self-supervised continual pre-training in both NLP and Vision is sufficient to mitigate forgetting without the use of any Continual Learning strategy. Other factors, like model depth, input modality and architecture type are not as crucial.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Hang Su
2. Jun Zhu
3. Liyuan Wang
4. Xingxing Zhang
5. Yunchao Wei
6. Guoliang Kang
7. Gengwei Zhang
8. Ling Chen

In Computer Vision and Pattern Recognition

1. Jun Zhu
2. Hang Su
3. Yunchao Wei
4. Liyuan Wang
5. Xingxing Zhang
6. Guoliang Kang
7. Gengwei Zhang
8. Ling Chen

In Multimodal Learning

1. Jun Zhu
2. Hang Su
3. Xingxing Zhang
4. Yunchao Wei

In Meta & Continual Learning

1. Xingxing Zhang
2. Liyuan Wang
3. Hang Su
4. Jun Zhu
5. Yunchao Wei
6. Gengwei Zhang
7. Ling Chen
8. Guoliang Kang

Citations in the corpus

• A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “An empirical study finds that self-supervised pre-training is more effective than supervised protocols for continual learning of vision-language models [216], consistent with the results for only visual tasks [28].”
• SLCA: Slow Learner with Classifier Alignment for Continual Learning on a Pre-trained Model (2023, background): “On the other hand, since the large amount of pre-training data are typically unlabeled and may also arrive incrementally, it seems more reasonable to use self-supervised pre-training than supervised pre-training [5, 43], also regarding that (upstream) continual learning in a self-supervised manner i”
• Continual Learning of Large Language Models: A Comprehensive Survey (2024, unknown): “Extensive studies have not only demonstrated the necessity of CPT for improved downstream performance [35, 71, 95, 96, 100, 184], but also shown that when distributional shifts are gradual [95, 278] or somewhat correlated [71], CPT can effectively help model generalize to unseen data.”