Mnemonics Training: Multi-Class Incremental Learning Without Forgetting

By Yaoyao Liu, Anan Liu, Yuting Su, Bernt Schiele, Qianru Sun. 2020

413 citations recorded by the source corpus.

Multi-Class Incremental Learning (MCIL) aims to learn new concepts by incrementally updating a model trained on previous concepts. However, there is an inherent trade-off to effectively learning new concepts without catastrophic forgetting of previous ones. To alleviate this issue, it has been proposed to keep around a few examples of the previous concepts but the effectiveness of this approach heavily depends on the representativeness of these examples. This paper proposes a novel and automatic framework we call mnemonics, where we parameterize exemplars and make them optimizable in an end-to-end manner. We train the framework through bilevel optimizations, i.e., model-level and exemplar-level. We conduct extensive experiments on three MCIL benchmarks, CIFAR-100, ImageNet-Subset and ImageNet, and show that using mnemonics exemplars can surpass the state-of-the-art by a large margin. Interestingly and quite intriguingly, the mnemonics exemplars tend to be on the boundaries between different classes.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “Besides, the old training samples can be continually adjusted to accommodate incremental changes, e.g., making them more representative [143] or challenging [144] for separation.”
Learnability and Algorithm for Continual Learning (2023, methodology): “The average forgetting rate after task t is Ft = ∑t−1 i=1 A i i − Ati (Liu et al., 2020b).”
Continual Learning by Modeling Intra-Class Variation (2022, methodology): “A number of methods (Wu et al., 2019; Hou et al., 2019; Liu et al., 2020; Douillard et al., 2020) take a path similar to iCaRL and utilize an extra model with a memory buffer to prevent forgetting.”
Continual Training of Language Models for Few-Shot Learning (2022, methodology): “…2019), language learning (Li et al., 2019), sentence embedding (Liu et al., 2019a), translation (Khayral-lah et al., 2018), cross-lingual modeling (Liu et al., 2020b), question answering (Greco et al., 2019) and text classiﬁcation (Ke et al., 2021a,b; Sun et al., 2020; Huang et al., 2021; Chuang…”
Class-Incremental Learning with Cross-Space Clustering and Controlled Transfer (2022, methodology): “Replay-based methods [42, 19, 13, 51, 6, 7, 33, 8, 30, 2, 47, 46, 54] store a subset of each previous task in a separate memory, and replay the tasks when learning a new one, to directly preserve the knowledge on those tasks.”
S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learning (2022, methodology): “Distillation-based methods (e.g., [40, 55, 22, 5, 67, 70, 62, 44, 49, 11]) apply the technique of knowledge distillation [21] to mitigate catastrophic forgetting.”
Novel Class Discovery without Forgetting (2022, background): “Regularization based methods [31,41,9,53,13,34] ensure that the”
Adaptive Orthogonal Projection for Batch and Online Continual Learning (2022, methodology): “In computer vision, several researchers used the term incremental learning to mean Class-IL as they learn like any other Class-IL method with multiple classes per task (Wu et al. 2019; Castro et al. 2018; Liu et al. 2020; Lee et al. 2019; Belouadah and Popescu 2019, 2020).”
Learning to Imagine: Diversify Memory for Incremental Learning using Unlabeled Data (2022, methodology): “For ImageNetSubset, we use the original RseNet-18 as our backbone following prior works [6,12,18,28,32,33].”
Constrained Few-shot Class-incremental Learning (2022, methodology): “To address the catastrophic forgetting issue in CIL, rehearsal methods store some old training data, and replay them while learning novel classes during the incremental stage [1, 3, 31, 32, 36, 39].”
On Generalizing Beyond Domains in Cross-Domain Continual Learning (2022, background): “[24] use bi-level optimization for synthesizing the memory, and a more optimal memory replay is expected compared to storing exemplars from training data.”
Memory Replay with Data Compression for Continual Learning (2022, neutral): “Another solution is feature replay: GFR (Liu et al., 2020a) learned a feature generator from a feature extractor to replay generated features, but the feature extractor suffered from catastrophic forgetting since it was incrementally updated.”
vCLIMB: A Novel Video Class Incremental Learning Benchmark (2022, background): “Although there is extensive literature on continual learning [1, 7, 8, 15, 17, 21, 25, 27, 31, 42], we restrict our review to a sample of the most widely adopted baselines.”