Subspace Regularizers for Few-Shot Class Incremental Learning

By Afra Feyza Akyürek, Ekin Akyürek, Derry Tanti Wijaya, Jacob Andreas. 2021

83 citations recorded by the source corpus.

Few-shot class incremental learning -- the problem of updating a trained classifier to discriminate among an expanded set of classes with limited labeled data -- is a key challenge for machine learning systems deployed in non-stationary environments. Existing approaches to the problem rely on complex model architectures and training procedures that are difficult to tune and re-use. In this paper, we present an extremely simple approach that enables the use of ordinary logistic regression classifiers for few-shot incremental learning. The key to this approach is a new family of subspace regularization schemes that encourage weight vectors for new classes to lie close to the subspace spanned by the weights of existing classes. When combined with pretrained convolutional feature extractors, logistic regression models trained with subspace regularization outperform specialized, state-of-the-art approaches to few-shot incremental image classification by up to 22% on the miniImageNet dataset. Because of its simplicity, subspace regularization can be straightforwardly extended to incorporate additional background information about the new classes (including class names and descriptions specified in natural language); these further improve accuracy by up to 2%. Our results show that simple geometric regularization of class representations offers an effective tool for continual learning.

Subfields: Artificial Intelligence, Cancer Research, Computer Vision and Pattern Recognition, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Han-Jia Ye
2. De-chuan Zhan
3. Da-Wei Zhou
4. Jun Zhu
5. Hang Su
6. Liyuan Wang
7. Xingxing Zhang
8. Keon-Hee Park

In Cancer Research

1. Han-Jia Ye
2. De-chuan Zhan
3. Dacheng Tao
4. Philip H. S. Torr
5. Jun Zhu
6. Gyeong-Moon Park

In Computer Vision and Pattern Recognition

1. Han-Jia Ye
2. De-chuan Zhan
3. Da-Wei Zhou
4. Jun Zhu
5. Hang Su
6. Liyuan Wang
7. Xingxing Zhang
8. Dacheng Tao

In Meta & Continual Learning

1. Han-Jia Ye
2. De-chuan Zhan
3. Da-Wei Zhou
4. Liyuan Wang
5. Xingxing Zhang
6. Hang Su
7. Jun Zhu
8. Gyeong-Moon Park

Citations in the corpus

• A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “Besides, auxiliary information such as semantic word vectors [262], [263] and sketch exemplars [264] can be incorporated to enrich the limited training samples.”
• S3C: Self-Supervised Stochastic Classifiers for Few-Shot Class-Incremental Learning (2023, background): “The works proposed in [9, 8, 1, 10, 48, 21, 32] focus on learning robust manifolds by regularizing feature space representations.”
• S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learning (2022, methodology): “In addition, there exist other kinds of methods like gradient-based methods (e.g., [56, 73, 16, 58, 61, 64]), regularization-based ones (e.g. [32, 74, 2, 1]), and expansion-based ones (e.g. [69, 34, 25]).”
• Few-Shot Class-Incremental Learning by Sampling Multi-Phase Tasks (2022, methodology): “Typical GFSL algorithms try to solve the problem by classifier weight generalization [72], [74], subspace regularization [75] and attention-based regularization [76].”
• LFS-GAN: Lifelong Few-Shot Image Generation (2023, background): “Previous studies on lifelong few-shot learning have primarily focused on discriminative tasks [1, 32, 21, 40, 38, 41].”
• A Survey on Few-Shot Class-Incremental Learning (2023, background): “In Aky¨urek et al. (2022), the authors propose a subspace regularization scheme that encourages the weight representation of new classes to be close to the subspace spanned by the weights of existing old classes.”
• Neural Collapse Inspired Feature-Classifier Alignment for Few-Shot Class Incremental Learning (2023, background): “However, the two goals rely on sophisticated loss functions or regularizers (Chen & Lee, 2021; Hersche et al., 2022; Aky ¨ urek et al., 2022), and are hard to attain simultaneously without qualification.”
• Pre-trained Vision and Language Transformers are Few-Shot Incremental Learners (2024, methodology): “Feature and feature space-based methods [2, 12, 23, 30, 41, 42] enable the model to adapt to new classes better and improve the generalization ability of feature extractors for new classes.”
• Warping the Space: Weight Space Rotation for Class-Incremental Few-Shot Learning (2023, background): “Moreover, we found that (Aky¨urek et al., 2022) provided the results with ResNet12 architecture as done in (Hersche et al., 2022), thus we also evaluate our 1 (Hersche et al., 2022) 76.37 70.94 66.36 62.64 59.31 56.02 53.14 51.04 48.87 C-FSCIL Mode 2 (Hersche et al., 2022) 76.45 71.23 66.71 63.01…”