Few-Shot Lifelong Learning

By Pratik Mazumder, Pravendra Singh, Piyush Rai. 2021

159 citations recorded by the source corpus.

Many real-world classification problems often have classes with very few labeled training samples. Moreover, all possible classes may not be initially available for training, and may be given incrementally. Deep learning models need to deal with this two-fold problem in order to perform well in real-life situations. In this paper, we propose a novel Few-Shot Lifelong Learning (FSLL) method that enables deep learning models to perform lifelong/continual learning on few-shot data. Our method selects very few parameters from the model for training every new set of classes instead of training the full model. This helps in preventing overfitting. We choose the few parameters from the model in such a way that only the currently unimportant parameters get selected. By keeping the important parameters in the model intact, our approach minimizes catastrophic forgetting. Furthermore, we minimize the cosine similarity between the new and the old class prototypes in order to maximize their separation, thereby improving the classification performance. We also show that integrating our method with self-supervision improves the model performance significantly. We experimentally show that our method significantly outperforms existing methods on the miniImageNet, CIFAR-100, and CUB-200 datasets. Specifically, we outperform the state-of-the-art method by an absolute margin of 19.27% for the CUB dataset.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “…[24], constructing an exemplar relation graph for knowledge distillation [250], selectively updating only the unimportant parameters [251] or stabilizing the important parameters [252], updating parameters within the ﬂat region of loss landscape [84], meta-learning of a good…”
S3C: Self-Supervised Stochastic Classifiers for Few-Shot Class-Incremental Learning (2023, methodology): “As is common in most FSCIL approaches [44,25], we propose to freeze the feature extractor and learn only the SC at each incremental step.”
Towards Label-Efficient Incremental Learning: A Survey (2023, background): “Few-shot Lifelong Learning [Mazumder et al., 2021] selects a few weights from the architecture to fine-tune perincremental learning task.”
Learning with Recoverable Forgetting (2022, positive): “FSLL [30] is proposed to perform on the few-shot setting by selecting very few parameters from the model.”
Few-Shot Class-Incremental Learning by Sampling Multi-Phase Tasks (2022, methodology): “FSLL [69] adopts the idea from CIL algorithms, which selects a few parameters to be updated at every incremental session to resist overﬁtting.”
Forward Compatible Few-Shot Class-Incremental Learning (2022, methodology): “To resist overﬁtting on few-shot inputs, FSLL [28] selects a few parameters to be updated at every incremental session.”
Overcoming Catastrophic Forgetting in Incremental Few-Shot Learning by Finding Flat Minima (2021, methodology): “The study of incremental few-shot learning has just started [47, 41, 60, 9, 8, 34, 59].”
Few-Shot Class-Incremental Learning for Named Entity Recognition (2022, methodology): “We also implement FSLL (Mazumder et al., 2021), a state-of-the-art method of few-shot class-incremental learning for image classification with metric learning.”
Warping the Space: Weight Space Rotation for Class-Incremental Few-Shot Learning (2023, methodology): “We compare our WaRP with FSLL (Mazumder et al., 2021), which also finetunes the model by selecting trainable parameters as ours, but in the original parameter space defined with the standard basis.”
SIN: Semantic Inference Network for Few-Shot Streaming Label Learning (2022, background): “Mazumder et al. [47] select a few parameters to learn novel classes rather than updating the whole network, which helps prevent overﬁtting.”
LFS-GAN: Lifelong Few-Shot Image Generation (2023, background): “The above example assumes that the rank r =2.”
Task Agnostic Representation Consolidation: a Self-supervised based Continual Learning Approach (2022, background): “(Zhang et al., 2020; Caccia et al., 2021; Mazumder et al., 2021; Su et al., 2020), relinquished pre-training altogether and employed auxiliary pretext task to boost task-agnostic learning.”
A Survey on Few-Shot Class-Incremental Learning (2023, methodology): “From the perspective of parameter feature space, Kim et al. (2023) proposed WaRP by fusing the advantages of F2M (SHI et al., 2021) for finding flat minimums of the loss function and FSLL (Mazumder et al., 2021) for parameter fine-tuning.”
Pre-trained Vision and Language Transformers are Few-Shot Incremental Learners (2024, methodology): “Prototype-based methods [6, 19, 39] aim to align prototypes with classifier weights to enhance the classification performance.”
On the Soft-Subnetwork for Few-shot Class Incremental Learning (2022, methodology): “FSLL [31] aims to search session-specific subnetworks while preserving weights for previous sessions for incremental few-shot learning.”
Semantics-Driven Generative Replay for Few-Shot Class Incremental Learning (2022, methodology): “FSLL [24], instead, is a regularizationbased method which handles overfitting and catastrophic forgetting by training only ‘unimportant’ model parameters in every session.”
SemGIF: A Semantics Guided Incremental Few-shot Learning Framework with Generative Replay (2021, background): “It is also worth mentioning that a related but different problem setup from IFSL is few-shot class incremental learning (FSCIL) [1, 19, 36, 46] consisting of a large number of incremental episodes with each stage accumulating knowledge from a small number of classes under limited supervision.”