Rehearsal-Free Continual Learning over Small Non-I.I.D. Batches

By Vincenzo Lomonaco, Davide Maltoni, Lorenzo Pellegrini. 2019

74 citations recorded by the source corpus.

Robotic vision is a field where continual learning can play a significant role. An embodied agent operating in a complex environment subject to frequent and unpredictable changes is required to learn and adapt continuously. In the context of object recognition, for example, a robot should be able to learn (without forgetting) objects of never before seen classes as well as improving its recognition capabilities as new instances of already known classes are discovered. Ideally, continual learning should be triggered by the availability of short videos of single objects and performed on-line on on-board hardware with fine-grained updates. In this paper, we introduce a novel continual learning protocol based on the CORe50 benchmark and propose two rehearsal-free continual learning techniques, CWR* and AR1*, that can learn effectively even in the challenging case of nearly 400 small non-i.i.d. incremental batches. In particular, our experiments show that AR1* can outperform other state-of-the-art rehearsal-free techniques by more than 15% accuracy in some cases, with a very light and constant computational and memory overhead across training batches.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

In Meta & Continual Learning

Citations in the corpus

ORBIT: A Real-World Few-Shot Dataset for Teachable Object Recognition (2021, methodology): “The ORBIT dataset can be used to explore a wide set of real-world recognition tasks from continual learning [27, 28] to video segmentation [25, 34, 29].”
CODA-Prompt: COntinual Decomposed Attention-Based Prompting for Rehearsal-Free Continual Learning (2022, background): “Other works have looked at rehearsal-free continual learning from an online “streaming” learning perspective using a frozen, pre-trained model [20, 40].”
DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning (2022, background): “Nevertheless, the dependence on rehearsal buffer has been criticised in the community [52, 12, 45, 30].”
Continual Normalization: Rethinking Batch Normalization for Online Continual Learning (2022, background): “However, the continual learning community stills mostly adopt the standard BN in their backbone network, and lack a systematic study regarding normalization layers (Lomonaco et al., 2020).”
An Empirical Investigation of the Role of Pre-training in Lifelong Learning (2021, background): “They make the point that in many scenarios (Lomonaco et al., 2019), it would be unrealistic to deploy agents with no pre-training in a LL setting.”
PLOP: Learning without Forgetting for Continual Semantic Segmentation (2020, background): “Longer Continual Learning: We argue that CSS experiments should push towards more steps [65, 45, 17, 6] to quantify the robustness of approaches w.”
Optimal Continual Learning has Perfect Memory and is NP-hard (2020, background): “As this implicit assumption is usually severely violated in practice, regularization-based CL algorithms often underperform, especially when the number of tasks is very large (van de Ven & Tolias, 2019; Lomonaco et al., 2019).”
Latent Replay for Real-Time Continual Learning (2019, methodology): “Despite these nuisances, in [6] the approaches denoted as CWR* and AR1* proved to be able”
Avalanche: an End-to-End Library for Continual Learning (2021, neutral): “Strategies Naive (Finetuning), CWR* [34], Replay, GDumb [43], Cumulative, LwF [30], GEM [35], A-GEM [6], EWC [27], Synaptic Intelligence [56], AR1 [36], Joint Training.”
Continual Learning in Deep Networks: an Analysis of the Last Layer (2021, result): “γ‖z‖ · cos(∠(zt, Ai)) + b = oi (5) We can note that the γ parameter is similar to the parameters of the batch-normalization [20], which has been shown to hurt training of intermediate layers in continual learning [34].”
A Closer Look at Rehearsal-Free Continual Learning * (2022, background): “Online Rehearsal-Free Continual Learning: Other works have looked at rehearsal-free continual learning from an online “streaming” learning perspective using a frozen, pretrained model [17,38].”
Continual Learning of Large Language Models: A Comprehensive Survey (2024, unknown): “Throughout training, models have limited or no access to previous data, posing a challenge in retaining past knowledge as optimization constraints from unseen previous data are absent during current-task learning [124, 135, 213].”
Class-Incremental Learning with Generative Classifiers (2021, background): “These drawbacks have sparked recent interest in ‘rehearsal-free’ continual learning [32], in which storing”
A TinyML Platform for On-Device Continual Learning With Quantized Latent Replays (2021, methodology): “On the Core50 dataset, the CL setting is regulated by the NICv2391 protocol [59].”
Tackling Catastrophic Forgetting and Background Shift in Continual Semantic Segmentation (2021, background): “We argue that CSS experiments should push towards more steps [59], [63], [91], [110] to quantify the robustness of approaches w.”
Prototype-based Incremental Few-Shot Segmentation (2020, methodology): “To achieve this, we take inspiration from continual learning works with non-i.i.d. data [25] and use batch-renorm (BR) [19].”