Continual Learning with Foundation Models: An Empirical Study of Latent Replay

By Oleksiy Ostapenko, Timothée Lesort, P. Rodr'iguez, Md Rifat Arefin, Arthur Douillard, Irina Rish, Laurent Charlin. 2022

79 citations recorded by the source corpus.

Rapid development of large-scale pre-training has resulted in foundation models that can act as effective feature extractors on a variety of downstream tasks and domains. Motivated by this, we study the efficacy of pre-trained vision models as a foundation for downstream continual learning (CL) scenarios. Our goal is twofold. First, we want to understand the compute-accuracy trade-off between CL in the raw-data space and in the latent space of pre-trained encoders. Second, we investigate how the characteristics of the encoder, the pre-training algorithm and data, as well as of the resulting latent space affect CL performance. For this, we compare the efficacy of various pre-trained models in large-scale benchmarking scenarios with a vanilla replay setting applied in the latent and in the raw-data space. Notably, this study shows how transfer, forgetting, task similarity and learning are dependent on the input data characteristics and not necessarily on the CL algorithms. First, we show that under some circumstances reasonable CL performance can readily be achieved with a non-parametric classifier at negligible compute. We then show how models pre-trained on broader data result in better performance for various replay sizes. We explain this with representational similarity and transfer properties of these representations. Finally, we show the effectiveness of self-supervised pre-training for downstream domains that are out-of-distribution as compared to the pre-training domain. We point out and validate several research directions that can further increase the efficacy of latent CL including representation ensembling. The diverse set of datasets used in this study can serve as a compute-efficient playground for further CL research. The codebase is available under https://github.com/oleksost/latent_CL.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Learnability and Algorithm for Continual Learning (2023, background): “Note that all baselines and ROW use the same architecture and the same pre-training model for fairness as using a pre-trained model improves the performance (Kim et al., 2022a; Ostapenko et al., 2022).”
First Session Adaptation: A Strong Replay-Free Baseline for Class-Incremental Learning (2023, methodology): “Current SOTA methods for class incremental learning start from a pre-trained backbone [13, 19, 35, 34] and then adapt the features at each session of continual learning.”
A Simple Baseline that Questions the Use of Pretrained-Models in Continual Learning (2022, background): “[18] studied foundation models and replay of frozen latent features to overcome catastrophic forgetting.”
Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters (2024, methodology): “Memory-based methods [30, 40, 47, 57, 59, 61, 64] retain the historical knowledge by storing them in a memory bank, which will be accessed and updated in incremental learning.”
Emergent and Predictable Memorization in Large Language Models (2023, background): “This can be understood as a form of continual learning in large language models [48] via implicit experience replay [38].”
A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPT (2023, background): “For some scenarios [253], it suffices to perform the continual learning on downstream tasks while keeping the pre-trained foundation model unchanged.”