Semantic Drift Compensation for Class-Incremental Learning

By Lu Yu, Bartłomiej Twardowski, Xialei Liu, Luis Herranz, Kai Wang, Yongmei Cheng, Shangling Jui, Joost van de Weijer. 2020

422 citations recorded by the source corpus.

Class-incremental learning of deep networks sequentially increases the number of classes to be classified. During training, the network has only access to data of one task at a time, where each task contains several classes. In this setting, networks suffer from catastrophic forgetting which refers to the drastic drop in performance on previous tasks. The vast majority of methods have studied this scenario for classification networks, where for each new task the classification layer of the network must be augmented with additional weights to make room for the newly added classes. Embedding networks have the advantage that new classes can be naturally included into the network without adding new weights. Therefore, we study incremental learning for embedding networks. In addition, we propose a new method to estimate the drift, called semantic drift, of features and compensate for it without the need of any exemplars. We approximate the drift of previous tasks based on the drift that is experienced by current task data. We perform experiments on fine-grained datasets, CIFAR100 and ImageNet-Subset. We demonstrate that embedding networks suffer significantly less from catastrophic forgetting. We outperform existing methods which do not require exemplars and obtain competitive results compared to methods which store exemplars. Furthermore, we show that our proposed SDC when combined with existing methods to prevent forgetting consistently improves results.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, methodology): “Other methods exploit the class-wise statistics of feature representations to obtain a balanced classifier, such as by imposing the representations to be transferable and invariant (SPB [453], IL2A [509], SSRE [512], FeTrIL [333]), or compensating explicitly the representation shifts (SDC [484], RER”
S3C: Self-Supervised Stochastic Classifiers for Few-Shot Class-Incremental Learning (2023, background): “Some of the recent works [47,41,46] have focused on non-exemplar based methods, with no access to exemplars from the old classes.”
RMM: Reinforced Memory Management for Class-Incremental Learning (2023, background): “…task-incremental learning—each phase corresponds to a task (dataset) that contains new data of all seen classes [7, 10, 19, 29, 43, 47, 57], or class-incremental learning (CIL)—each phase contains data of a new set of classes, i.e., classes are unseen [4, 6, 18, 25, 31, 37, 40, 41, 49, 52, 55–57].”
FeTrIL: Feature Translation for Exemplar-Free Class-Incremental Learning (2022, methodology): “FeTrIL shares the idea of using class prototypes with [45, 47, 48, 49].”
On the Effectiveness of Lipschitz-Driven Rehearsal in Continual Learning (2022, neutral): “Indeed, due to the limited size of the training set, competitive performance can only be achieved if each task can benefit from the initialization [20, 88].”
Continual Learning by Modeling Intra-Class Variation (2022, methodology): “The last type of method, activation-based regularization (Li & Hoiem, 2017; Rebuffi et al., 2017; Hou et al., 2019; Wu et al., 2019; Yu et al., 2020), leverages the information from the activations obtained from past tasks using strategies similar to knowledge distillation (Hinton et al., 2015).”
Anti-Retroactive Interference for Lifelong Learning (2022, background): “E-MAS-SDC proposed by [32] estimates the drift of previous tasks during the training of new tasks to make semantic drift compensation.”
Few-Shot Class-Incremental Learning from an Open-Set Perspective (2022, neutral): “As real-world classification problems typically exhibit class imbalance or long-tailed data distribution, some methods have explored deep metric learning for incremental and long-tail tasks [22,15,12].”
S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learning (2022, methodology): “Distillation-based methods (e.g., [40, 55, 22, 5, 67, 70, 62, 44, 49, 11]) apply the technique of knowledge distillation [21] to mitigate catastrophic forgetting.”
Online Continual Learning with Contrastive Vision Transformer (2022, background): “Recent studies have focused on a more practical Task-free protocol, which evaluates the network on all classes observed during training without requiring the identity of the task [3,81,8,95,37,90,13].”
Continual Learning through Retrieval and Imagination (2022, background): “For Tiny ImageNet and CIFAR-10, we follow (Buzzega et al. 2020; Rebuffi et al. 2017) (Buzzega et al. 2020; Yu et al. 2020).”
Transfer without Forgetting (2022, neutral): “Most notably, on Split CUB-200 oEWC shows performance lower than Finetune on both Task- and Class-IL.”
General Incremental Learning with Domain-aware Categorical Representations (2022, background): “It is also worth noting that while continual learning without data memory has attracted much attention in literature [32, 40, 42], these methods typically perform less effectively than those with a limited data memory for storing old examples [9, 37].”
Few-Shot Class-Incremental Learning by Sampling Multi-Phase Tasks (2022, methodology): “There are extensive efforts in resisting forgetting from different perspectives, e.g., knowledge distillation [14], [15], parameter consolidation [16], [17] and post-tuning [18], [19].”
Self-Sustaining Representation Expansion for Non-Exemplar Class-Incremental Learning (2022, background): “Recently, some works [36, 38, 44] focus on a challenging but practical non-exemplar class-incremental learning problem, where no past data can be stored due to equipment limits or privacy security.”
Representation Compensation Networks for Continual Semantic Segmentation (2022, methodology): “While others [31, 78, 95] utilize high-dimensional information, self-training and model adaptation to overcome this problem.”
Overcoming Catastrophic Forgetting in Incremental Few-Shot Learning by Finding Flat Minima (2021, methodology): “The second term enforces the class prototypes fixed within such region, which is designed to solve the prototype drift problem [54, 8] (the class prototypes change after updating the network) in later incremental learning sessions such that the saved base class prototypes can be directly used for ev”