Class-Incremental Learning by Knowledge Distillation with Adaptive Feature Consolidation

By Minsoo Kang, Jaeyoo Park, Bohyung Han. 2022

257 citations recorded by the source corpus.

We present a novel class incremental learning approach based on deep neural networks, which continually learns new tasks with limited memory for storing examples in the previous tasks. Our algorithm is based on knowledge distillation and provides a principled way to maintain the representations of old models while adjusting to new tasks effectively. The proposed method estimates the relationship between the representation changes and the resulting loss increases incurred by model updates. It minimizes the upper bound of the loss increases using the representations, which exploits the estimated importance of each feature map within a backbone model. Based on the importance, the model restricts updates of important features for robustness while allowing changes in less critical features for flexibility. This optimization strategy effectively alleviates the notorious catastrophic forgetting problem despite the limited accessibility of data in the previous tasks. The experimental results show significant accuracy improvement of the proposed algorithm over the existing methods on the standard datasets. Code is available.11https://github.com/kminsoo/AFC

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, methodology): “Feature Space LUCIR [164], PODNet [100], TPCIL [403], PCL [170], AANets [270], DER [468], DDE [171], GeoDL [387], PASS [510], Co2L [57], AFC [199], SP-CIL [456], ELI [192], CwD [380], CSCCT [22], FOSTER [424], FASP [300], CLS-ER [18]”
First Session Adaptation: A Strong Replay-Free Baseline for Class-Incremental Learning (2023, background): “Thus, recent works either focus on fixing the softmax classifier [4, 1, 44] or deploy nearest class mean classifiers (NCM) as an alternative [25, 5, 11].”
Deep Class-Incremental Learning: A Survey (2023, background): “Feature Distillation [93], [94], [95], [96], [97], [98], [99], [100]”
Class-Incremental Learning: A Survey (2023, background): “There are several ways to expand representation ability, and we divide them into three sub-groups, i.e. , neuron expansion , backbone expansion , and prompt expansion .”
LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning (2023, background): “Although some recent works explore the combination of different approaches [2, 29, 54] or new strategies [74, 57, 14], our benchmark aims to provide an in-depth analysis of these three basic lifelong learning directions to reveal their pros and cons on robot learning tasks.”
Elastic Feature Consolidation for Cold Start Exemplar-free Incremental Learning (2024, background): “Hou et al. (2019); Douillard et al. (2020); Kang et al. (2022) combine cosine normalization and distillation losses with exemplars to mitigate forgetting.”
Incremental Nuclei Segmentation from Histopathological Images via Future-class Awareness and Compatibility-inspired Distillation (2024, background): “The constraints can be applied on features [10, 12, 18], the output logits [4, 11, 21], the weights [1, 2, 19], or the gradients [6, 15].”