A Simple Baseline that Questions the Use of Pretrained-Models in Continual Learning

By P. Janson, Rahaf Aljundi, Mohamed Elhoseiny. 2022

65 citations recorded by the source corpus.

With the success of pretraining techniques in representation learning, a number of continual learning methods based on pretrained models have been proposed. Some of these methods design continual learning mechanisms on the pre-trained representations and only allow minimum updates or even no updates of the backbone models during the training of continual learning. In this paper, we question whether the complexity of these models is needed to achieve good performance by comparing them to a simple baseline that we designed. We argue that the pretrained feature extractor itself can be strong enough to achieve a competitive or even better continual learning performance on Split-CIFAR100 and CoRe 50 benchmarks. To validate this, we conduct a very simple baseline that 1) use the frozen pretrained model to extract image features for every class encountered during the continual learning stage and compute their corresponding mean features on training data, and 2) predict the class of the input based on the nearest neighbor distance between test samples and mean features of the classes; i.e., Nearest Mean Classifier (NMC). This baseline is single-headed, exemplar-free, and can be task-free (by updating the means continually). This baseline achieved 88.53% on 10-Split-CIFAR-100, surpassing most state-of-the-art continual learning methods that are all initialized using the same pretrained transformer model. We hope our baseline may encourage future progress in designing learning systems that can continually add quality to the learning representations even if they started from some pretrained weights.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Evaluation & Benchmarks, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, methodology): “Representative strategies include prompting [402], [457], [458] or carefully fine-tuning [303], [389], [504] the backbone, saving prototypes to rectify the predictions [188], [335], [504], incorporating transfer learning techniques [43], [114], [330], etc.”
Resurrecting Old Classes with New Data for Exemplar-Free Continual Learning (2024, methodology): “Following recent studies [11, 16, 26], we explore using class prototypes with an NCM [36] classifier and show that a simple baseline of logits distillation [24] with an NCM classifier often outperforms existing EFCIL methods in the small-start setting.”
Embedding World Knowledge into Tabular Models: Towards Best Practices for Embedding Pipeline Design (2026, unknown): “Together, our results offer guidance for building more effective embedding pipelines for tabular prediction and lay the groundwork for future research on compressible embedding models [20, 21, 22], embedding-specific downstream models [23], or alternative serialization options [4].”