Encoders and Ensembles for Task-Free Continual Learning

By Murray Shanahan, Christos Kaplanis, Jovana Mitrovi'c. 2021

26 citations recorded by the source corpus.

We present an architecture that is effective for continual learning in an especially demanding setting, where task boundaries do not exist or are unknown, and where classes have to be learned online (with each example presented only once). To obtain good performance under these constraints, while mitigating catastrophic forgetting, we exploit recent advances in contrastive, self-supervised learning, allowing us to use a pre-trained, general purpose image encoder whose weights can be frozen, which precludes forgetting. The pre-trained encoder also greatly simplifies the downstream task of classification, which we solve with an ensemble of very simple classifiers. Collectively, the ensemble exhibits much better performance than any individual classifier, an effect which is amplified through specialisation and competitive selection. We assess the performance of the encoders-and-ensembles architecture on standard continual learning benchmarks, where it outperforms prior state-of-the-art by a large margin on the hardest problems, as well as in less familiar settings where the data distribution changes gradually or the classes are presented one at a time.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

Learning to Prompt for Continual Learning (2021, methodology): “where there is only a single test set available, we report the final test accuracy following the common protocol [30, 52].”
Model Zoo: A Growing Brain That Learns Continually (2021, background): “…(Zenke et al., 2017), Permuted-MNIST (Kirkpatrick et al., 2017), Split-CIFAR10 (Zenke et al., 2017), Split-CIFAR100 ∗ (Zenke et al., 2017), Coarse-CIFAR100 (Rosenbaum et al., 2017; Yoon et al., 2019; Shanahan et al., 2021) and Split-miniImagenet (Vinyals et al., 2016; Chaudhry et al., 2019b).”
Online Task-free Continual Learning with Dynamic Sparse Distributed Memory (2022, methodology): “DSDM outperforms Ensemble while requiring much less data.”
Three types of incremental learning (2022, unknown): “From here on, we will use the term task only to refer to a context when it is always clear to the learning algorithm when a sample belongs to that context (as is the case with task-incremental learning).”
Online Continual Learning For Interactive Instruction Following Agents (2024, background): “Moreover, such an update often exploits task boundary information that might not always be available, especially in the cases of streamed data without explicit task boundaries (Shanahan et al., 2021; Koh et al., 2023).”