Continuum: Simple Management of Complex Continual Learning Scenarios

By Arthur Douillard, Timothée Lesort. 2021

42 citations recorded by the source corpus.

Continual learning is a machine learning sub-field specialized in settings with non-iid data. Hence, the training data distribution is not static and drifts through time. Those drifts might cause interferences in the trained model and knowledge learned on previous states of the data distribution might be forgotten. Continual learning's challenge is to create algorithms able to learn an ever-growing amount of knowledge while dealing with data distribution drifts. One implementation difficulty in these field is to create data loaders that simulate non-iid scenarios. Indeed, data loaders are a key component for continual algorithms. They should be carefully designed and reproducible. Small errors in data loaders have a critical impact on algorithm results, e.g. with bad preprocessing, wrong order of data or bad test set. Continuum is a simple and efficient framework with numerous data loaders that avoid researcher to spend time on designing data loader and eliminate time-consuming errors. Using our proposed framework, it is possible to directly focus on the model design by using the multiple scenarios and evaluation metrics implemented. Furthermore the framework is easily extendable to add novel settings for specific needs.

Subfields: Artificial Intelligence, Civil and Structural Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Civil and Structural Engineering

Citations in the corpus

Simpler is Better: off-the-shelf Continual Learning Through Pretrained Backbones (2022, methodology): “We used the timm [25] library to fetch the pretrained models where all the models have been trained on ImageNet [23] and the continuum [5] library to create the incremental setting for 5 datasets, namely CIFAR10/100, Core50, OxfordFlow-ers102 and TinyImageNet200.”
Continual Learning via Local Module Composition (2021, background): “The rapid growth of continual learning has lead researchers to work on empirical studies [20, 58, 56], surveys [32, 42, 55, 66, 67] as well as CL-specific software [72, 22, 59].”
Regularization Shortcomings for Continual Learning (2019, methodology): “We experiment with the “MNIST-Fellowship” scenario proposed in [7].”
Avalanche: an End-to-End Library for Continual Learning (2021, background): “Continuum [12] focuses on data loading and processing, and could be related with the functionalities provided by the benchmark module of Avalanche.”
Continual Learning with Foundation Models: An Empirical Study of Latent Replay (2022, methodology): “A recent papers on natural language processing (NLP) indicate that SSL of large transformer models (Devlin et al., 2018; Reddy et al., 2021; Douillard et al., 2021) on vast amounts of pre-training data results in models that, without further training, generalize well to new tasks from few examples…”
Understanding Continual Learning Settings with Data Distribution Drift Analysis (2021, background): “The creation of a well-designed dataset for continual learning remains an important challenge for the community notably because they are many types of application scenarios and many data types (Douillard and Lesort, 2021).”
Class-Incremental Learning with Generative Classifiers (2021, methodology): “As in [14, 31], we use the images from eight of these environments for training and the others for testing.”
Pretrained Language Model in Continual Learning: A Comparative Study (2022, background): “…researchers to work on empirical studies (De Lange et al., 2019; Lesort et al., 2021b), surveys (Hadsell et al., 2020; Khetarpal et al., 2020; Lesort et al., 2021a; Mundt et al., 2020; 2021) as well as CL-speciﬁc software (Normandin et al., 2021; Douillard & Lesort, 2021; Lomonaco et al., 2021).”
Scaling the Number of Tasks in Continual Learning (2022, methodology): “Existing continual learning settings, such as class-incremental or domain incremental scenarios van de Ven and Tolias [2019], Douillard and Lesort [2021], Normandin et al. [2021], do evaluate catastrophic forgetting in short sequences of unique tasks but they do not evaluate knowledge accumulation…”
Three types of incremental learning (2022, unknown): “However, in recent years, several resources have been created that provide—or enable the generation of—more realistic context sets well suited for domain-incremental learning 24,41,62–64 .”
Tackling Catastrophic Forgetting and Background Shift in Continual Semantic Segmentation (2021, background): “which is not the case in continual segmentation where multiple shifts [71], [72], [90], happen in the training tasks, different from the testing tasks.”
CoMFormer: Continual Learning in Semantic and Panoptic Segmentation (2022, methodology): “We start from the widely adopted continual semantic segmentation benchmark defined in [4] and we extend it to the newly proposed continual panoptic segmentation using a modified Continuum library [18].”
Sequoia: A Software Framework to Unify Continual Learning Research (2021, methodology): “Supervised Learning environments Sequoia supports most of the datasets traditionally used in continual supervised learning research, through its use of the Continuum package (Douillard & Lesort, 2021).”
Continual Learning in Deep Networks: an Analysis of the Last Layer (2021, background): “5, how the scenarios are created with the continuum library [13].”