Distilling Causal Effect of Data in Class-Incremental Learning

By Xinting Hu, Kaihua Tang, Chunyan Miao, Xiansheng Hua, Hanwang Zhang. 2021

213 citations recorded by the source corpus.

We propose a causal framework to explain the catastrophic forgetting in Class-Incremental Learning (CIL) and then derive a novel distillation method that is orthogonal to the existing anti-forgetting techniques, such as data replay and feature/label distillation. We first 1) place CIL into the framework, 2) answer why the forgetting happens: the causal effect of the old data is lost in new training, and then 3) explain how the existing techniques mitigate it: they bring the causal effect back. Based on the causal framework, we propose to distill the Colliding Effect between the old and the new data, which is fundamentally equivalent to the causal effect of data replay, but without any cost of replay storage. Thanks to the causal effect analysis, we can further capture the Incremental Momentum Effect of the data stream, removing which can help to retain the old effect overwhelmed by the new data effect, and thus alleviate the forgetting of the old class in testing. Extensive experiments on three CIL benchmarks: CIFAR-100, ImageNet-Sub&Full, show that the proposed causal effect distillation can improve various state-of-the-art CIL methods by a large margin (0.72%–9.06%). 1

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Meta & Continual Learning, Causal Inference & Discovery

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Meta & Continual Learning

In Causal Inference & Discovery

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, methodology): “In this regard, replay of the old training samples is able to impose an end-to-end effect, while feature and label distillations of the new training samples are limited to the output side with a biased distribution [171].”
RMM: Reinforced Memory Management for Class-Incremental Learning (2023, methodology): “Based on the CIL protocol [13, 18, 20, 30], D0 contains half of the classes of the whole dataset, e.”
Learning with Recoverable Forgetting (2022, positive): “Hu et al. [21] derive a distillation method to retain the old effect overwhelmed by the new data effect, and thus alleviate the forgetting of the old class in testing.”
Learning to Imagine: Diversify Memory for Incremental Learning using Unlabeled Data (2022, methodology): “For ImageNetSubset, we use the original RseNet-18 as our backbone following prior works [6,12,18,28,32,33].”
Class-Incremental Learning by Knowledge Distillation with Adaptive Feature Consolidation (2022, methodology): “For comparisons, we report average incremental accuracy [37]—the average of the accuracies over seen classes across all incremental stages, where the results of other methods except GeoDL [44], DDE [17], and SSIL [2] are retrieved from [12] if not specified otherwise.”
Federated Class-Incremental Learning (2022, methodology): “[17] uses the causal effect on knowledge distillation to rectify class imbalance.”
Self-Sustaining Representation Expansion for Non-Exemplar Class-Incremental Learning (2022, methodology): “Rehearsal-based methods [1, 7, 12, 22, 33] preserve exemplars of fixed memory size to maintain the distribution of old classes in the incremental phases, and adopt the distillation skills to retain the discriminative features of the old task.”
Memory Replay with Data Compression for Continual Learning (2022, positive): “In particular, memory replay of representative old training samples (referred to as memory replay) can generally achieve the best performance in class-incremental learning (Liu et al., 2021a; Hu et al., 2021) and in numerous other continual learning scenarios, such as audio tasks (Ehret et al.”
Mimicking the Oracle: An Initial Phase Decorrelation Approach for Class Incremental Learning (2021, methodology): “Two classic setups of Incremental Learning are Class In-cremental Learning (CIL) [1,12,13,19,20,22,26] and Task Incremental Learning (TIL) [2, 16, 21, 24, 28, 30, 31].”
Incremental Learning in Semantic Segmentation from Image Labels (2021, background): “Even if in image classification it has been exhaustively studied [1, 11, 20, 22, 24, 30, 38, 41, 54, 57, 68], in semantic segmentation it is still in its early stages [8, 9, 21, 33, 43, 45–47].”
Overcoming Catastrophic Forgetting in Incremental Few-Shot Learning by Finding Flat Minima (2021, methodology): “Previous research mainly includes multi-class incremental learning [4, 38, 22, 33, 54, 51] and multi-task incremental learning [21, 31, 42].”
Recent Advances of Continual Learning in Computer Vision: An Overview (2021, neutral): “[56] found that the knowledge distillation based approaches do not have a consistent causal effect compared to end-to-end feature learning.”
Co-Transport for Class-Incremental Learning (2021, background): “Class-incremental learning (CIL) is now a popular topic in the machine learning community [6, 9, 15, 25, 39, 47, 62, 68, 70].”
Adaptive Aggregation Networks for Class-Incremental Learning (2020, background): “, each phase has the data of a new set of classes coming from the identical dataset [4, 16, 18, 25, 34, 48, 53].”
Technical Report for ICCV 2021 Challenge SSLAD-Track3B: Transformers Are Better Continual Learners (2022, methodology): “To overcome catastrophic forgetting in artificial neural networks, three types of methods have been explored in the past few years, replay [15, 3, 6, 20, 10], regularization [9, 19, 8, 4, 17, 13, 5, 7, 16], and expand [22, 14, 1, 21].”
Semantics-Driven Generative Replay for Few-Shot Class Incremental Learning (2022, methodology): “Likewise, methods like BiC [41], LUCIR [11], PODNet [6], and AANets [20] attempted to better exploit the memory buffer from variety of ways, such as by mitigating the data imbalance between consecutive tasks [12].”
FFNB: Forgetting-Free Neural Blocks for Deep Continual Learning (2021, methodology): “,T10 correspond to classes [51-100] while T0 is the pretraining task involving classes [1-50]).”
Class-Incremental Learning: A Survey (2023, background): “There are several ways to expand representation ability, and we divide them into three sub-groups, i.e. , neuron expansion , backbone expansion , and prompt expansion .”