Semantic-aware Knowledge Distillation for Few-Shot Class-Incremental Learning

By Ali Cheraghian, Shafin Rahman, Pengfei Fang, Soumava Kumar Roy, Lars Petersson, Mehrtash Harandi. 2021

233 citations recorded by the source corpus.

Few-shot class incremental learning (FSCIL) portrays the problem of learning new concepts gradually, where only a few examples per concept are available to the learner. Due to the limited number of examples for training, the techniques developed for standard incremental learning cannot be applied verbatim to FSCIL. In this work, we introduce a distillation algorithm to address the problem of FSCIL and propose to make use of semantic information during training. To this end, we make use of word embeddings as semantic information which is cheap to obtain and which facilitate the distillation process. Furthermore, we propose a method based on an attention mechanism on multiple parallel embeddings of visual data to align visual and semantic vectors, which reduces issues related to catastrophic forgetting. Via experiments on MiniImageNet, CUB200, and CIFAR100 dataset, we establish new state-of-the-art results by outperforming existing approaches.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “Besides, auxiliary information such as semantic word vectors [262], [263] and sketch exemplars [264] can be incorporated to enrich the limited training samples.”
S3C: Self-Supervised Stochastic Classifiers for Few-Shot Class-Incremental Learning (2023, methodology): “As in [44] [9], we propose to freeze the already learnt feature extractor, since the self-supervision has ensured that it will generalize well to previously unseen classes.”
Towards Label-Efficient Incremental Learning: A Survey (2023, methodology): “, 2021], clustering [Chen and Lee, 2020; Cheraghian et al., 2021; Akyürek et al., 2022; Zhou et al., 2022] or the architecture [Mazumder et al.”
Few-Shot Class-Incremental Learning from an Open-Set Perspective (2022, neutral): “We compare our method with the SOTA methods [17,1,11,19,4,5,23] on three datasets.”
Incremental Few-Shot Semantic Segmentation via Embedding Adaptive-Update and Hyper-class Representation (2022, methodology): “Most current incremental methods [7, 11, 24, 49] use parameters or embedding vectors to represent category knowledge from a cognitive-inspired perspective, which addresses catastrophic forgetting by knowledge representation with new-class updating scheme.”
S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learning (2022, methodology): “Distillation-based methods (e.g., [40, 55, 22, 5, 67, 70, 62, 44, 49, 11]) apply the technique of knowledge distillation [21] to mitigate catastrophic forgetting.”
Few-Shot Class-Incremental Learning by Sampling Multi-Phase Tasks (2022, methodology): “distillation [70] considers using auxiliary word embedding of few-shot instances to boost model updating.”
Doodle It Yourself: Class Incremental Learning by Drawing a Few Sketches (2022, methodology): “Very few methods are present to tackle the FSCIL problem like, pseudo incremental learning [70], knowledge distillation [9, 13], neural-gas network [54].”
Forward Compatible Few-Shot Class-Incremental Learning (2022, background): “Semantic-aware knowledge distillation [10] treats the word embedding as auxiliary information, and builds knowledge distillation terms to resist forgetting.”
Representation Compensation Networks for Continual Semantic Segmentation (2022, neutral): “In the more challenging scenario without any replay, many methods explore regularization to maintain old knowledge, including knowledge distillation [11,19,22,29,52,70,75], adversarial training [30,90], vanilla regularization [44,49,57,67,68,81,97,100] and so on.”
Overcoming Catastrophic Forgetting in Incremental Few-Shot Learning by Finding Flat Minima (2021, background): “Incremental few-shot learning [47, 41, 60, 9, 8] aims to incrementally learn from very few samples.”
Subspace Regularizers for Few-Shot Class Incremental Learning (2021, background): “Existing approaches to this problem either prioritize novel class adaptation (Ren et al., 2019; Yoon et al., 2020; Chen & Lee, 2021; Cheraghian et al., 2021) or reducing forgetting in old classes (Tao et al., 2020b).”
Co-Transport for Class-Incremental Learning (2021, background): “Class-incremental learning (CIL) is now a popular topic in the machine learning community [6, 9, 15, 25, 39, 47, 62, 68, 70].”
On Learning the Geodesic Path for Incremental Learning (2021, methodology): “A distillation loss for incremental learning is extended to the feature space with a measurement using the Euclidean distance proposed in [6,17].”
Continually Learning Self-Supervised Representations with Projected Functional Regularization (2021, background): “This can be done by either regularizing the weight space (constraining important parameters) [57, 60] or the functional space (constraining predictions or intermediate features) [13, 19, 31].”
MetaFSCIL: A Meta-Learning Approach for Few-Shot Class Incremental Learning (2022, methodology): “In addition, the aforesaid approaches involve hand-engineered heuristics (e.g. saving exemplars [3, 7], decoupled learning [32], prototype refinement [33]) which are defective compared to the learned solutions [6].”