Few-Shot Class-Incremental Learning via Relation Knowledge Distillation

By Songlin Dong, Xiaopeng Hong, Xiaoyu Tao, Xinyuan Chang, Xing Wei, Yihong Gong. 2021

224 citations recorded by the source corpus.

In this paper, we focus on the challenging few-shot class incremental learning (FSCIL) problem, which requires to transfer knowledge from old tasks to new ones and solves catastrophic forgetting. We propose the exemplar relation distillation incremental learning framework to balance the tasks of old-knowledge preserving and new-knowledge adaptation. First, we construct an exemplar relation graph to represent the knowledge learned by the original network and update gradually for new tasks learning. Then an exemplar relation loss function for discovering the relation knowledge between different classes is introduced to learn and transfer the structural information in relation graph. A large number of experiments demonstrate that relation knowledge does exist in the exemplars and our approach outperforms other state-of-the-art class-incremental learning methods on the CIFAR100, miniImageNet, and CUB200 datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “The extremely limited training samples exacerbate the overfitting of previously-learned representations to subsequent tasks, which can be alleviated by recent work such as preserving the topology of representations (TOPIC [404]), constructing an exemplar relation graph for knowledge distillation (ER”
S3C: Self-Supervised Stochastic Classifiers for Few-Shot Class-Incremental Learning (2023, neutral): “The works in [11, 34, 44] used graph-based networks for old classes’ knowledge retention.”
Few-Shot Class-Incremental Learning from an Open-Set Perspective (2022, neutral): “Most papers about FSCIL [4,25,7,23,27,5] follow the task setting proposed by Tao et al.”
Learning with Recoverable Forgetting (2022, methodology): “Knowledge distillation could also be a reliable method to transfer knowledge from old data to new data, and there are also some distillation-based works [6, 16, 38, 7] for solving the coming new data in life-long learning setting.”
Few-Shot Class-Incremental Learning by Sampling Multi-Phase Tasks (2022, positive): “Exemplar Relation Graph [68] maintains a graph to represent the class-wise relationship, which is utilized for knowledge distillation.”
Doodle It Yourself: Class Incremental Learning by Drawing a Few Sketches (2022, methodology): “Very few methods are present to tackle the FSCIL problem like, pseudo incremental learning [70], knowledge distillation [9, 13], neural-gas network [54].”
Class Incremental Learning for Video Action Classification (2021, background): “There are three main directions to overcome catastrophic forgetting: (1) To regularize the important parameters of the old model when training new data [9, 10, 11, 12], (2) To retain some old samples for knowledge distillation technology [13, 14, 15, 16, 17], (3) To generate old samples based on gen”
A Survey on Few-Shot Class-Incremental Learning (2023, methodology): “Unlike CIL based on individual knowledge distillation (Park et al., 2019), Dong et al. (2021) applied graph distillation techniques to FSCIL for the first time.”
MetaFSCIL: A Meta-Learning Approach for Few-Shot Class Incremental Learning (2022, methodology): “In addition, the aforesaid approaches involve hand-engineered heuristics (e.g. saving exemplars [3, 7], decoupled learning [32], prototype refinement [33]) which are defective compared to the learned solutions [6].”
Class-Incremental Learning: A Survey (2023, background): “…[84], [85], [86], [87], [88] Feature Distillation [89], [90], [91], [92], [93], [94], [95], [96], [97], [98] Relational Distillation [27], [99], [100], [101], [102], [103] § 3.6 Model Rectify Feature Rectify [104], [105], [106], [107], [108], [109] Logit Rectify [83], [89], [110], [111] Weight…”
Deep Class-Incremental Learning: A Survey (2023, background): “Relational Distillation [101], [102], [103], [104], [105]”
Neural Collapse Inspired Feature-Classifier Alignment for Few-Shot Class Incremental Learning (2023, methodology): “…(CIL) (Cauwenberghs & Poggio, 2000; Li & Hoiem, 2017; Rebufﬁ et al., 2017), FSCIL only has a few novel-classes and training data in each incremental session (Tao et al., 2020b; Dong et al., 2021), which increases the tendency of overﬁtting on novel classes (Snell et al., 2017; Sung et al., 2018).”