M2KD: Multi-model and Multi-level Knowledge Distillation for Incremental Learning

By Peng Zhou, Long Mai, Jianming Zhang, Ning Xu, Zuxuan Wu, Larry S. Davis. 2019

58 citations recorded by the source corpus.

Incremental learning targets at achieving good performance on new categories without forgetting old ones. Knowledge distillation has been shown critical in preserving the performance on old classes. Conventional methods, however, sequentially distill knowledge only from the last model, leading to performance degradation on the old classes in later incremental learning steps. In this paper, we propose a multi-model and multi-level knowledge distillation strategy. Instead of sequentially distilling knowledge only from the last model, we directly leverage all previous model snapshots. In addition, we incorporate an auxiliary distillation to further preserve knowledge encoded at the intermediate feature levels. To make the model more memory efficient, we adapt mask based pruning to reconstruct all previous models with a small memory footprint. Experiments on standard incremental learning benchmarks show that our method preserves the knowledge on old classes better and improves the overall performance over standard distillation techniques.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

In Meta & Continual Learning

Citations in the corpus

Memory Efficient Continual Learning with Transformers (2022, methodology): “The idea is adopted in CL and incremental learning domain to maintain the responses of the network unchanged on the old tasks whilst updating it with new training samples in different ways [52, 7, 31, 70].”
DyTox: Transformers for Continual Learning with DYnamic TOken eXpansion (2021, neutral): “A different TAB per task leads to better results (56% v.s. 52% in “Last” accuracy) because the network can be more diverse with each TAB specialized to its associated task.”
PLOP: Learning without Forgetting for Continual Semantic Segmentation (2020, background): “Among existing distillation schemes based on intermediate features [17, 70, 56, 16, 78, 29], POD [17] consists in matching global statistics at different feature levels between the old and current models.”
Knowledge Distillation: A Survey (2020, methodology): “…2019; Yang et al. 2020c; Tang et al. 2019; Hu et al. 2018; Sun et al. 2019; Nakashole and Flauger 2017; Jiao et al. 2020; Wang et al. 2018d; Zhou et al. 2019a; Sanh et al. 2019; Turc et al. 2019; Arora et al. 2019; Clark et al. 2019; Kim and Rush 2016; Mou et al. 2016; Liu et al. 2019f;…”
Faster ILOD: Incremental Learning for Object Detectors based on Faster RCNN (2020, methodology): “(Zhou et al., 2019) proposed a multi-model distillation method called M2KD which directly matches the category outputs of the current model with those of the corresponding old models.”
ScaIL: Classifier Weights Scaling for Class Incremental Learning (2020, methodology): “Unlike existing IL algorithms which use distillation [5, 7, 10, 12, 22, 34], only classification loss is used.”
Maintaining Discrimination and Fairness in Class Incremental Learning (2019, methodology): “Knowledge distillation (KD) [12] is another strategy, which has also been widely used in this ﬁeld [5, 20, 39].”
Knowledge Distillation for Incremental Learning in Semantic Segmentation (2019, background): “ preserving existing knowledge. For this reason, incremental learning is gaining wide relevance among the scientiﬁc community and has already been explored in image classiﬁcation and object detection [1, 2, 3, 4, 5, 6, 7, 8, 9]. On the other hand, incremental learning in dense labeling tasks, such a”
Incremental Learning Techniques for Semantic Segmentation (2019, methodology): “In [37] the current model distills the knowledge from pruned versions of all previous model snapshots.”
A Comprehensive Study of Class Incremental Learning Algorithms for Visual Tasks (2020, methodology): “In spite of this observation, knowledge distillation is commonly used to tackle catastrophic forgetting [10, 16, 20, 23, 35, 41, 65].”
Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks (2020, methodology): “As a result, many new KD methods [9], [51], [56], [57], [80], [101], [114], [118], [131], [134], [139], [147], [155], [161], [174], [175], [182], [185], [186], [191], [195], [196], [201], [205], [208], [210], [214], [219], [231], [232], [234], [249], [252], [258], [263], [267], [272], [275] have bee”
Memory Efficient Continual Learning for Neural Text Classification (2022, methodology): “The idea is adopted in CL and incremental learning domain to maintain the responses of the network unchanged on the old tasks whilst updating it with new training samples in different ways (Shmelkov et al., 2017; Castro et al., 2018; Li & Hoiem, 2017; Zhou et al., 2019).”
Initial Classifier Weights Replay for Memoryless Class Incremental Learning (2020, methodology): “We build on existing works to: exploit information from the initial state of classifiers [5, 31]; ensure fairness for the predictions associated to past and new classes [4, 13, 29]; use vanilla FT as a backbone to train deep models [1, 5] .”
More Classifiers, Less Forgetting: A Generic Multi-classifier Paradigm for Incremental Learning (2020, background): “Upon LwF, many approaches have been proposed to improve the regularization based on knowledge distillation [43,7,50,22,49].”
Small-Task Incremental Learning (2020, unknown): “Employing a distillation-based loss to constrain the evolution of the representation has also resulted in leading results [12,32,35,4].”
PODNet: Pooled Outputs Distillation for Small-Tasks Incremental Learning (2020, unknown): “Employing a distillation-based loss to constrain the evolution of the representation has also resulted in leading results [13,35,37,5].”