IL2M: Class Incremental Learning With Dual Memory

By Eden Belouadah, Adrian Daniel Popescu. 2019

393 citations recorded by the source corpus.

This paper presents a class incremental learning (IL) method which exploits fine tuning and a dual memory to reduce the negative effect of catastrophic forgetting in image recognition. First, we simplify the current fine tuning based approaches which use a combination of classification and distillation losses to compensate for the limited availability of past data. We find that the distillation term actually hurts performance when a memory is allowed. Then, we modify the usual class IL memory component. Similar to existing works, a first memory stores exemplar images of past classes. A second memory is introduced here to store past class statistics obtained when they were initially learned. The intuition here is that classes are best modeled when all their data are available and that their initial statistics are useful across different incremental states. A prediction bias towards newly learned classes appears during inference because the dataset is imbalanced in their favor. The challenge is to make predictions of new and past classes more comparable. To do this, scores of past classes are rectified by leveraging contents from both memories. The method has negligible added cost, both in terms of memory and of inference complexity. Experiments with three large public datasets show that the proposed approach is more effective than a range of competitive state-of-the-art methods.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “IL2M [140] and SNCL [79] recover statistics of feature representations (e.g., mean and covariance) on the basis of experience replay.”
S3C: Self-Supervised Stochastic Classifiers for Few-Shot Class-Incremental Learning (2023, methodology): “Several state-of-art-works [38,45,2,12,3] proposed different techniques to address the class imbalance problem in CIL like rescaling scores or balanced finetuning of classifiers, etc.”
RMM: Reinforced Memory Management for Class-Incremental Learning (2023, background): “…task-incremental learning—each phase corresponds to a task (dataset) that contains new data of all seen classes [7, 10, 19, 29, 43, 47, 57], or class-incremental learning (CIL)—each phase contains data of a new set of classes, i.e., classes are unseen [4, 6, 18, 25, 31, 37, 40, 41, 49, 52, 55–57].”
Long-Tailed Class Incremental Learning (2022, methodology): “Replay-based methods are very effective and recall knowledge from previous tasks by maintaining a small memory of samples [2,4,7,13,32,41], representations [12], or synthetic data [35].”
Meta-Learning with Less Forgetting on Large-Scale Non-Stationary Task Distributions (2022, neutral): “Many works mitigate catastrophic forgetting during the learning process [37,14,48,68,31,42,20,2,10,3,64].”
Class-Incremental Learning with Cross-Space Clustering and Controlled Transfer (2022, methodology): “Replay-based methods [42, 19, 13, 51, 6, 7, 33, 8, 30, 2, 47, 46, 54] store a subset of each previous task in a separate memory, and replay the tasks when learning a new one, to directly preserve the knowledge on those tasks.”
Novel Class Discovery without Forgetting (2022, background): “Another kind of approach either stores or generates exemplar images for all the tasks introduced to the model so far and replays them while learning a new task [41,34,6,27].”
Adaptive Orthogonal Projection for Batch and Online Continual Learning (2022, methodology): “In computer vision, several researchers used the term incremental learning to mean Class-IL as they learn like any other Class-IL method with multiple classes per task (Wu et al. 2019; Castro et al. 2018; Liu et al. 2020; Lee et al. 2019; Belouadah and Popescu 2019, 2020).”
Few-Shot Class-Incremental Learning by Sampling Multi-Phase Tasks (2022, background): “Other works consider saving embeddings instead of raw images [38], using generative models for data rehearsal [39], task-wise adaptation [40], and output normalization [18], [41], [42].”
Energy-based Latent Aligner for Incremental Learning (2022, methodology): “In replay based methods, typically a small subset of exemplars is stored to recall and retain representations useful for earlier tasks [6, 20, 24, 32, 41].”
Forward Compatible Few-Shot Class-Incremental Learning (2022, methodology): “Other methods rehearsal former instances to overcome forgetting [6, 48, 60, 66].”
Representation Compensation Networks for Continual Semantic Segmentation (2022, background): “To solve this problem, many work [4, 5,11,47,77] propose to review knowledge by rehearsal-based mechanism.”
On Generalizing Beyond Domains in Cross-Domain Continual Learning (2022, background): “On the other hand, there has been a surge of efforts to devise machine learning based algorithms to build more intelligent models and mitigate the aforementioned challenges from two perspectives, namely continual learning [2, 4, 26, 32] and domain generalSequence of Domains Sequence of Tasks”
Class-Incremental Continual Learning Into the eXtended DER-Verse (2022, neutral): “Several other works combine distillation with rehearsal : iCaRL [11] distills the responses of the model at the previous task boundary, learning latent representations to be used in a nearest mean-of-exemplars classiﬁer; EtEIL [40], LUCIR [31] and BiC [41] focus on contrasting the prediction bias th”
Mimicking the Oracle: An Initial Phase Decorrelation Approach for Class Incremental Learning (2021, methodology): “To mitigate this dilemma, some methods have been proposed to maintain a good balance between old and new classes [4, 19, 32, 33].”
DyTox: Transformers for Continual Learning with DYnamic TOken eXpansion (2021, background): “Consequently, the model is better calibrated, which is an important attribute of continual model [4, 75, 81].”
Online Continual Learning Via Candidates Voting (2021, background): “However, they also limit the model’s ability to learn new task and can even harm the performance if the teacher model used by distillation [17] is not learned on large balanced data [5].”
Recent Advances of Continual Learning in Computer Vision: An Overview (2021, neutral): “They used selective memory retrieval, which is shown to be more effective compared to random retrieval in continual analogical reasoning.”
Generalized and Incremental Few-Shot Learning by Explicit Learning and Calibration without Forgetting (2021, methodology): “We prefer these two measures rather than using only the joint performance in joint space due to the imbalance of the number of classes [19, 3, 49] with |CB | ≫ |CN | (e.”
Co-Transport for Class-Incremental Learning (2021, background): “[5, 63] simply normalize the linear layer with weight clipping to maintain fairness between old and new classes.”