Incremental Few-Shot Learning with Attention Attractor Networks

By Mengye Ren, Renjie Liao, Ethan Fetaya, Richard S. Zemel. 2018

189 citations recorded by the source corpus.

Machine learning classifiers are often trained to recognize a set of pre-defined classes. However, in many applications, it is often desirable to have the flexibility of learning additional concepts, with limited data and without re-training on the full training set. This paper addresses this problem, incremental few-shot learning, where a regular classification network has already been trained to recognize a set of base classes, and several extra novel classes are being considered, each with only a few labeled examples. After learning the novel classes, the model is then evaluated on the overall classification performance on both base and novel classes. To this end, we propose a meta-learning model, the Attention Attractor Network, which regularizes the learning of novel classes. In each episode, we train a set of new weights to recognize novel classes until they converge, and we show that the technique of recurrent back-propagation can back-propagate through the optimization process and facilitate the learning of these parameters. We demonstrate that the learned attractor network can help recognize novel classes while remembering old classes without the need to review the original training set, outperforming various baselines.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Meta & Continual Learning

Citations in the corpus

RMM: Reinforced Memory Management for Class-Incremental Learning (2023, background): “…task-incremental learning—each phase corresponds to a task (dataset) that contains new data of all seen classes [7, 10, 19, 29, 43, 47, 57], or class-incremental learning (CIL)—each phase contains data of a new set of classes, i.e., classes are unseen [4, 6, 18, 25, 31, 37, 40, 41, 49, 52, 55–57].”
Few-Shot Class-Incremental Learning by Sampling Multi-Phase Tasks (2022, methodology): “Typical GFSL algorithms try to solve the problem by classifier weight generalization [72], [74], subspace regularization [75] and attention-based regularization [76].”
Overcoming Catastrophic Forgetting in Incremental Few-Shot Learning by Finding Flat Minima (2021, background): “Incremental few-shot learning [47, 41, 60, 9, 8] aims to incrementally learn from very few samples.”
Subspace Regularizers for Few-Shot Class Incremental Learning (2021, methodology): “In this section we describe the experiment setup for the single-session evaluation, ( T = 1) , and compare our approach to state-of-the-art XtarNet (Yoon et al., 2020), as well as Ren et al. (2019), Gidaris & Komodakis (2018) and Qi et al. (2018).”
Generalized and Incremental Few-Shot Learning by Explicit Learning and Calibration without Forgetting (2021, methodology): “In Table 1 the results show not only better performance than trivially skipping the second phase but also outperform the previous state-of-the-art [11].”
Self-Promoted Prototype Refinement for Few-Shot Class-Incremental Learning (2021, background): “These works aim to quickly adapt to novel classes from few training data while not forgetting the base classes [21] on which it was trained.”
Few-Shot Class-Incremental Learning via Relation Knowledge Distillation (2021, background): “Recently, some FSL works (Gidaris and Komodakis 2018; Ren et al. 2019) attempt to learn a uniﬁed model capable of recognizing both the base and novel classes while those are not able to keep learning new tasks because they rely on the base class for sampling meta-learning tasks.”
Incremental Few-shot Text Classification with Multi-round New Classes: Formulation, Dataset and System (2021, neutral): “In Ren et al. (2019 Qi et al. (2018), both Gidaris and Komodakis (2018) and Ren et al. (2019) build connections be-tween the representations of base classes and the new classes.”
Semantic-aware Knowledge Distillation for Few-Shot Class-Incremental Learning (2021, methodology): “[27] notes that the method of recurrent back-propagation can back-propagate within the optimization process and helps the learning of novel tasks.”
Few-Shot Lifelong Learning (2021, methodology): “Recently some methods have also explored the loss of performance in the non few-shot classes due to the techniques used to beneﬁt the few-shot classes (Gidaris and Komodakis 2018; Ren et al. 2019).”
GP-Tree: A Gaussian Process Classifier for Few-Shot Incremental Learning (2021, background): “…this field can be categorized according to three types: (i) Regularization based approaches impose regularization methods on the network to maintain past knowledge (Goodfellow et al., 2014; Kirkpatrick et al., 2017; Lee et al., 2017; Chaudhry et al., 2018; Schwarz et al., 2018; Ren et al., 2019).”
A Two-phase Prototypical Network Model for Incremental Few-shot Relation Classification (2020, background): “Currently, several related works (Qi et al., 2018; Gidaris and Komodakis, 2018; Xiang et al., 2019; Ren et al., 2019) are proposed in computer vision field and they concentrate on image classification task.”
Few-Shot Class-Incremental Learning (2020, background): “Recently, some FSL research works attempt to learn a model capable of recognizing both the base and novel classes [9, 33].”
Incremental Meta-Learning via Indirect Discriminant Alignment (2020, methodology): “Similarly, [27] used recurrent back-propagation to train a set of new weights to achieve good overall classification on both old and new classes.”
Large-Scale Long-Tailed Recognition in an Open World (2019, background): “The few-shot learning without forgetting [13] and incremental few-shot learning [39] attempt to remedy this issue by leveraging the duality between features and classifiers’ weights [36, 35].”