Towards Robust Evaluations of Continual Learning

By Sebastian Farquhar, Yarin Gal. 2018

329 citations recorded by the source corpus.

Experiments used in current continual learning research do not faithfully assess fundamental challenges of learning continually. Instead of assessing performance on challenging and representative experiment designs, recent research has focused on increased dataset difficulty, while still using flawed experiment set-ups. We examine standard evaluations and show why these evaluations make some continual learning approaches look better than they are. We introduce desiderata for continual learning evaluations and explain why their absence creates misleading comparisons. Based on our desiderata we then propose new experiment designs which we demonstrate with various continual learning approaches and datasets. Our analysis calls for a reprioritization of research effort by the community.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

Toward Understanding Catastrophic Forgetting in Continual Learning (2019, background): “In fact, some previous work showed that task similarity helps improve performance in the context of transfer learning [2, 3, 35], while some others claimed that task dissimilarity could help continual learning [13] although their discussion was more related to the permuted MNIST setting.”
Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselines (2018, background): “This scenario is argued to be easier since the selection of output head is given by the task identity [10].”
Continual Learning via Sequential Function-Space Variational Inference (2023, background): “Our evaluation pays attention to two factors important in the assessment of continual-learning methods: the use of task identifiers when making predictions, and the use of a coresets of data points to summarize past tasks (Farquhar and Gal, 2018).”
On the Effectiveness of Lipschitz-Driven Rehearsal in Continual Learning (2022, methodology): “the latter, TIL has been criticized as a less challenging and realistic benchmark [26, 74, 4]; we therefore conduct our main experiments on state-of-the-art CL models in the CIL setting1.”
Disentangling Transfer in Continual Reinforcement Learning (2022, background): “The popular CL desiderata include reducing the forgetting on previous tasks and increasing the forward transfer on the new tasks, i.e. speeding up the learning by reusing knowledge from previous tasks [12, 46].”
Transfer without Forgetting (2022, background): “The former scenario is a relaxation of the latter, as it provides the model with the task identifier of each sample at test time; for this reason, we focus our evaluation mainly on the ClassIL protocol, highlighted as a more realistic and challenging benchmark [20,3].”
Learning Fast, Learning Slow: A General Continual Learning Method based on Complementary Learning System (2022, methodology): “There are a plethora of evaluation protocols in the CL literature, each of which biases the evaluation towards a certain approach (Farquhar & Gal, 2018; Mi et al., 2020; Shim et al., 2020; van de Ven & Tolias, 2019).”
Class-Incremental Continual Learning Into the eXtended DER-Verse (2022, neutral): “These methods can be effective for short sequences of tasks but usually fail to scale to complex problems [12], [32].”
Learning to Prompt for Continual Learning (2021, background): “The limitations of prior work bring up critical questions in continual learning [13, 16]: (1) Whether the form of episodic memory can go beyond buffering past data to more intelligent and succinct episodic memory system? (2) How”
Learning curves for continual learning in neural networks: Self-knowledge transfer and forgetting (2021, methodology): “The continual learning in the NTK regime belongs to so-called single-head setting (Farquhar & Gal, 2018), and it allows the model to revisit the previous classes (target functions) in subsequent tasks.”
DyTox: Transformers for Continual Learning with DYnamic TOken eXpansion (2021, background): “This avoids optimizing for the number of tasks, which defeats the purpose of continual learning [7].”
Continual Learning via Local Module Composition (2021, background): “Unfortunately, in many realistic CL scenarios task identities are unavailable at test time [23, 35, 14].”
Continual Learning on Noisy Data Streams via Self-Purified Replay (2021, methodology): “We explicitly ground our experiment setting based on the recent suggestions for robust evaluation in continual learing [1, 18, 79] as follows.”
Online Continual Learning with Natural Distribution Shifts: An Empirical Study with Visual Data (2021, background): “Since historical data are not cached for training, these methods often suffer from catastrophic forgetting, especially given a long data stream [6].”
Continual semi-supervised learning through contrastive interpolation consistency (2021, methodology): “Continual Learning is an umbrella term encompassing several slightly yet meaningfully different experimental settings [7, 8].”
Continual Learning in the Teacher-Student Setup: Impact of Task Similarity (2021, methodology): “Many methods for combating catastrophic interference have been proposed, often taking the form of regularisation, architecture expansion, and/or replay (Parisi et al., 2019; Farquhar & Gal, 2018).”
Model Zoo: A Growing Brain That Learns Continually (2021, methodology): “This indicates that replaying data from past tasks is beneﬁcial (Robins, 1995), even if replay may not conform to certain stylistic formulations of continual learning in the literature (Farquhar and Gal, 2019a; Kaushik et al., 2021).”
Same State, Different Task: Continual Reinforcement Learning without Interference (2021, methodology): “Furthermore single-headed networks are often used as more difficult CL scenarios when studying methods to mitigate forgetting (van de Ven and Tolias 2018; Farquhar and Gal 2018).”
Continual World: A Robotic Benchmark For Continual Reinforcement Learning (2021, methodology): “In this work we expand on previous discussions on the topic [42, 15, 29, 43] and introduce a new benchmark, Continual World.”
ACAE-REMIND for Online Continual Learning with Compressed Feature Replay (2021, background): “However, it has been shown that data-focused methods are vulnerable to domain shifts [2] and prior-focused methods might be not sufficient to restrict the optimization process to keep acceptable performances on previous tasks [9].”