Overcoming Catastrophic Forgetting in Incremental Few-Shot Learning by Finding Flat Minima

By Guangyuan Shi, Jiaxin Chen, Wenlong Zhang, Li-Ming Zhan, Xiao-Ming Wu. 2021

219 citations recorded by the source corpus.

This paper considers incremental few-shot learning, which requires a model to continually recognize new categories with only a few examples provided. Our study shows that existing methods severely suffer from catastrophic forgetting, a well-known problem in incremental learning, which is aggravated due to data scarcity and imbalance in the few-shot setting. Our analysis further suggests that to prevent catastrophic forgetting, actions need to be taken in the primitive stage -- the training of base classes instead of later few-shot learning sessions. Therefore, we propose to search for flat local minima of the base training objective function and then fine-tune the model parameters within the flat region on new tasks. In this way, the model can efficiently learn new classes while preserving the old ones. Comprehensive experimental results demonstrate that our approach outperforms all prior state-of-the-art methods and is very close to the approximate upper bound. The source code is available at https://github.com/moukamisama/F2M.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “As for optimizing an updatable backbone, F2M [389] searches for flat local minima in the pre-training stage, and then learns incremental tasks within the flat region.”
S3C: Self-Supervised Stochastic Classifiers for Few-Shot Class-Incremental Learning (2023, background): “The works proposed in [9,8,1,10,48,21,32] focus on learning robust manifolds by regular-izing feature space representations.”
CoSCL: Cooperation of Small Continual Learners is Stronger than a Big One (2022, positive): “Flatness of Loss Landscape provides a conceptual explanation of generalization for deep neural networks, which is recently introduced to understand catastrophic forgetting in continual learning [31,42,11,6,25].”
Constrained Few-shot Class-incremental Learning (2022, neutral): “Session (s) 1 2 3 4 5 6 7 8 9 No. of classes |C̃(s)| 60 65 70 75 80 85 90 95 100 Mode/ Work Feature extractor AL-MML [48] ResNet-18 64.10 55.88 47.07 45.16 40.11 36.38 33.96 31.55 29.37 Semantic KD∗ [6] ResNet-18 <64 <57 <51 <46 <43 <41 <39 <37 <35 VAE∗ [7] ResNet-18 <62 <58 <57 <52 <51 <49 <46 <45 ”
Neural Collapse Inspired Feature-Classifier Alignment for Few-Shot Class Incremental Learning (2023, methodology): “As a comparison, we pre-assign and ﬁx an optimal feature-classiﬁer alignment, and then train a model towards the same neural collapse optimality in each session to avoid target conﬂict. base session as a feature extractor (Zhang et al., 2021; Hersche et al., 2022; Aky¨urek et al., 2022).”
Semantics-Driven Generative Replay for Few-Shot Class Incremental Learning (2022, background): “Recently, F2M [32] proposed to tackle FSCIL by searching for flat local minima of the base training objective function and subsequently fine-tuning the model parameters within the flat region.”
Class-Incremental Learning: A Survey (2023, methodology): “Similarly, Neural Architecture Search (NAS) [176] is also adopted to find the optimal structure for each of the sequential tasks [61].”
On the Soft-Subnetwork for Few-shot Class Incremental Learning (2022, methodology): “Furthermore, we compare other FSCIL methods such as iCaRL [37], Rebalance [19], TOPIC [43], IDLVQ-C [5], and F2M [42].”
GKEAL: Gaussian Kernel Embedded Analytic Learning for Few-Shot Class Incremental Task (2023, methodology): “For instance, on CIFAR-100 F2M [22] requires 240 epochs for base training while the CEC [31], MetaFSCIL [3] and Entropy-reg [14] needs 100.”
A Survey on Few-Shot Class-Incremental Learning (2023, methodology): “In the majority of FSCIL experiments (Cheraghian et al., 2021c; Chi et al., 2022; Liu et al., 2022b; Peng et al., 2022; SHI et al., 2021; Tao et al., 2020; Yang et al., 2022; Zhang et al., 2021; Zheng and Zhang, 2021; Zhou et al., 2022b; Zhu et al., 2021a), the three image classification datasets…”
Warping the Space: Weight Space Rotation for Class-Incremental Few-Shot Learning (2023, methodology): “The performance of these baselines are mostly taken from (Zhang et al., 2021; Shi et al., 2021) for fair comparisons.”
A Model or 603 Exemplars: Towards Memory-Efficient Class-Incremental Learning (2022, background): “The research about CIL has attracted much interest [67, 32, 42] in the machine learning field.”
Pre-trained Vision and Language Transformers are Few-Shot Incremental Learners (2024, methodology): “Feature and feature space-based methods [2, 12, 23, 30, 41, 42] enable the model to adapt to new classes better and improve the generalization ability of feature extractors for new classes.”