Class-Incremental Continual Learning Into the eXtended DER-Verse

By Matteo Boschini, Lorenzo Bonicelli, Pietro Buzzega, Angelo Porrello, Simone Calderara. 2022

200 citations recorded by the source corpus.

The staple of human intelligence is the capability of acquiring knowledge in a continuous fashion. In stark contrast, Deep Networks forget catastrophically and, for this reason, the sub-field of Class-Incremental Continual Learning fosters methods that learn a sequence of tasks incrementally, blending sequentially-gained knowledge into a comprehensive prediction. This work aims at assessing and overcoming the pitfalls of our previous proposal Dark Experience Replay (DER), a simple and effective approach that combines rehearsal and Knowledge Distillation. Inspired by the way our minds constantly rewrite past recollections and set expectations for the future, we endow our model with the abilities to i) revise its replay memory to welcome novel information regarding past data ii) pave the way for learning yet unseen classes. We show that the application of these strategies leads to remarkable improvements; indeed, the resulting method – termed eXtended-DER (X-DER) – outperforms the state of the art on both standard benchmarks (such as CIFAR-100 and miniImageNet) and a novel one here introduced. To gain a better understanding, we further provide extensive ablation studies that corroborate and extend the findings of our previous research (e.g., the value of Knowledge Distillation and flatter minima in continual learning setups). We make our results fully reproducible; the codebase is available at https://github.com/aimagelab/mammoth.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, methodology): “Data Space iCaRL [356], GSS [17], Mnemonics [272], TPCIL [403], GDumb [340], DER++ [46], RMM [271], HAL [63], MRDC [431], CSI [217], X-DER [41]”
A soft nearest-neighbor framework for continual semi-supervised learning (2022, methodology): “We denote PseudoER as an additional two-stage baseline method, which is a combination of semi-supervised learning (PAWS) and CL (ER) methods.”
On the Effectiveness of Lipschitz-Driven Rehearsal in Continual Learning (2022, positive): “Previous CL works proposed investigating the generalization capabilities of a CL model by evaluating the flatness of its attained minima [15, 12].”
Transfer without Forgetting (2022, methodology): “This method has hence inspired several works: DER [8] and its extension XDER [6] also store past model responses and pin them as an additional teaching signal.”
Recent Advances of Continual Learning in Computer Vision: An Overview (2021, neutral): “They then proposed a method named Augmented Box Replay to handle this problem, and a loss function named attentive Region-of-Interest distillation loss to further reduce catastrophic forgetting.”
CaSpeR: Latent Spectral Regularization for Continual Learning (2023, methodology): “rehearsal-based methods: Experience Replay with Asymmetric Cross-Entropy (ER-ACE) [11], Incremental Classifier and Representation Learning (iCaRL) [38], Dark Experience Replay (DER++) [10], eXtended-DER (XDER)(3) [7] and Pooled Outputs Distillation Network (PODNet) [19].”
Learning to Predict Gradients for Semi-Supervised Continual Learning (2022, methodology): “Except for the aforementioned methods, the task incremental learning problem and class incremental learning problem can be solved by Dark Experience Replay (DER) [28] and eXtended-DER (X-DER) [29] simultaneously.”
Online Continual Learning For Interactive Instruction Following Agents (2024, methodology): “To address this, Buzzega et al. (2020); Boschini et al. (2022a) propose to store logits of past models for knowledge distillation, reducing storage costs while maintaining learning efficacy.”
Locality Sensitive Sparse Encoding for Learning World Models Online (2024, background): “Among continual learning methods with neural networks, rehearsal-based ones are closer to the FTL objec-tive and usually demonstrate strong performance (Chaudhry et al., 2019; Boschini et al., 2022).”