Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselines

By Yen-Chang Hsu, Yen-Cheng Liu, Zsolt Kira. 2018

401 citations recorded by the source corpus.

Continual learning has received a great deal of attention recently with several approaches being proposed. However, evaluations involve a diverse set of scenarios making meaningful comparison difficult. This work provides a systematic categorization of the scenarios and evaluates them within a consistent framework including strong baselines and state-of-the-art methods. The results provide an understanding of the relative difficulty of the scenarios and that simple baselines (Adagrad, L2 regularization, and naive rehearsal strategies) can surprisingly achieve similar performance to current mainstream methods. We conclude with several suggestions for creating harder evaluation scenarios and future research directions. The code is available at this https URL

Subfields: Artificial Intelligence, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Meta & Continual Learning

Citations in the corpus

LoRA: Low-Rank Adaptation of Large Language Models (2021, background): “Specifically, preforming the usual weight decay on A and B is similar to decaying back to the pre-trained weights, which has been studied as a potentially effective form of regularization against “catastrophic forgetting” [16, 18].”
A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “CIL [169], [425] {Dt, t}t∈T ; p(Xi) ̸= p(Xj) and Yi ∩ Yj = ∅ for i ̸= j {p(Xt)}t∈T ; t is unavailable”
CODA-Prompt: COntinual Decomposed Attention-Based Prompting for Rehearsal-Free Continual Learning (2022, background): “Class-incremental continual learning is challenging because the learner must support classification across all classes seen up to task n [25] (i.”
Online Continual Learning with Contrastive Vision Transformer (2022, methodology): “We consider a strict evaluation setting [38,75] for online continual learning, including task-aware protocol [56] and task-free protocol [39].”
CoSCL: Cooperation of Small Continual Learners is Stronger than a Big One (2022, positive): “We use a WideResNet-28-2 architecture to perform the task incremental learning experiments on CIFAR-100-RS, following a widelyused implementation code [15].”
Continual Learning through Retrieval and Imagination (2022, methodology): “We consider a strict evaluation setting (Hsu et al. 2018), which models the sequence of tasks following three scenarios: Task Incremental Learning ( Task-IL ) splits the training samples into partitions of tasks, which requires task identities to select corresponding classifiers at inference time;…”
Continual Learning with Guarantees via Weight Interval Constraints (2022, methodology): “To verify the empirical usefulness of our method, we test it in three standard continual learning scenarios (Hsu et al., 2018; van de Ven & Tolias, 2019): incremental task, incremental domain, and incremental class.”
Doodle It Yourself: Class Incremental Learning by Drawing a Few Sketches (2022, methodology): “Distillation [29] and memory-based [21] methods are more effective than regularisation-based ones [23] in the CIL setting.”
The Challenges of Continuous Self-Supervised Learning (2022, positive): “This idea is inspired by experience replay [39] commonly used in reinforcement learning [3, 49, 72] and supervised continual learning [31, 68].”
Architecture Matters in Continual Learning (2022, background): “Thus, Split CIFAR-100 and Split ImageNet-1K belong to the so-called task incremental learning setting, whereas Rotated MNIST belongs to domain incremental learning (Hsu et al., 2018).”
vCLIMB: A Novel Video Class Incremental Learning Benchmark (2022, methodology): “Consistent with image class incremental learning [14], the regularization-based methods EWC [17] and MAS [1] lag significantly behind replay-based methods, regardless of the difficulty of the dataset or the number of tasks.”
Wide Neural Networks Forget Less Catastrophically (2021, background): “Alleviating catastrophic forgetting has attracted a lot of attention lately, and many interesting solutions are proposed to partly overcome the issue (e.g., Toneva et al., 2019; Nguyen et al., 2019; Hsu et al., 2018; Li et al., 2019; Wallingford et al., 2020).”
Class-Incremental Learning for Action Recognition in Videos (2021, neutral): “However, these approaches empirically present poor generalization performance in class incremental learning scenarios as reported in [16, 39].”
AlterSGD: Finding Flat Minima for Continual Learning by Alternative Training (2021, background): “The penalty coefﬁcient λ reg represents the intensity of the penalty, which has a great impact on the performance of regularization methods for continual learning [14].”
Always Be Dreaming: A New Approach for Data-Free Class-Incremental Learning (2021, methodology): “The class-incremental learning setting is challenging because no task indexes are provided to the learner during inference and the learner must support classification across all classes seen up to task n [23].”
Rehearsal revealed: The limits and merits of revisiting samples in continual learning (2021, neutral): “2 k and 2 k) in the MNIST setup [9, 44, 45”
Gradient Projection Memory for Continual Learning (2021, methodology): “For permuted MNIST, we evaluate and compare our algorithm in ‘single-head’ setting (Hsu et al., 2018; Farquhar & Gal, 2018) where all tasks share the final classifier layer and inference is performed without task hint.”