Class-incremental Learning via Deep Model Consolidation

By Junting Zhang, Jie Zhang, Dawei Li, Serafettin Tasci, Larry Heck, Heming Zhang, C.-C. Jay Kuo. 2019

390 citations recorded by the source corpus.

Deep neural networks (DNNs) often suffer from "catastrophic forgetting" during incremental learning (IL) — an abrupt degradation of performance on the original set of classes when the training objective is adapted to a newly added set of classes. Existing IL approaches tend to produce a model that is biased towards either the old classes or new classes, unless with the help of exemplars of the old data. To address this issue, we propose a class-incremental learning paradigm called Deep Model Consolidation (DMC), which works well even when the original training data is not available. The idea is to first train a separate model only for the new classes, and then combine the two individual models trained on data of two distinct set of classes (old classes and new classes) via a novel double distillation training objective. The two existing models are consolidated by exploiting publicly available unlabeled auxiliary data. This overcomes the potential difficulties due to unavailability of original training data. Compared to the state-of-the-art techniques, DMC demonstrates significantly better performance in image classification (CIFAR-100 and CUB-200) and object detection (PASCAL VOC 2007) in the single-headed IL setting.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, methodology): “Some approaches exploit the unlabeled in-the-wild data to distill the old and new models into a shared model, in order to bridge potential non co-occurrence (BNC [96]) and achieve a better stability-plasticity tradeoff (DMC [493]).”
Towards Label-Efficient Incremental Learning: A Survey (2023, background): “Deep Model Consolidation [Zhang et al., 2020] is one of the earliest to leverage unlabeled auxiliary data to mitigate forgetfulness.”
Balancing Stability and Plasticity through Advanced Null Space in Continual Learning (2022, neutral): “Regularization-based methods can be divided into two classes: one is to distill knowledge from the previous model which is trained on the previous tasks [14, 15, 23, 36, 52, 59]; another is to explicitly use a regularizer to penalize the update of important parameters of previous tasks, preventing t”
Adaptive Orthogonal Projection for Batch and Online Continual Learning (2022, background): “The inaccessibility of the old data could be due to unrecorded legacy data, proprietary data, and data privacy, e.g., in federated learning (Zhang et al. 2020).”
Learning to Imagine: Diversify Memory for Incremental Learning using Unlabeled Data (2022, methodology): “A few previous works including GD [16] and DMC [37], use unlabeled data to assist training.”
Meta-attention for ViT-backed Continual Learning (2022, methodology): “The regularizationbased methods [21, 23, 26, 27, 34, 50, 52] utilize the knowledge distillation technique [15], special regularization terms to avoid catastrophic forgetting.”
Memory Efficient Continual Learning with Transformers (2022, methodology): “[71] where two individual image classification models trained on image data of two distinct set of classes (old classes and new classes) are consolidated into one single model that can classify all classes.”
Bridging Non Co-occurrence with Unlabeled In-the-wild Data for Incremental Object Detection (2021, positive): “Recently, several incremental learning methods [40, 27, 4, 10, 20, 31] are proposed to overcome catastrophic forgetting in the object detection task.”
Continual semi-supervised learning through contrastive interpolation consistency (2021, background): “proposed the Deep Model Consolidation framework [13]; in it, a new model is first specialized on each new encountered task,”
Continual Learning by Using Information of Each Class Holistically (2021, background): “The inaccessibility of the old task data could be due to many reasons, e.g., unrecorded legacy data, proprietary data, and privacy concerns such as in federated learning where the model can be shared from one party to another but the data must be kept private (Zhang et al. 2020).”
Split-and-Bridge: Adaptable Class Incremental Learning within a Single Neural Network (2021, background): “Another type of parameter isolation is to temporarily learn a new task in an extra network and then combine it into the previously learned model, which covers P&C (Schwarz et al. 2018), DR (Hou et al. 2018), DMC (Zhang et al. 2020), and GD (Lee et al. 2019).”
Self-Supervised Training Enhances Online Continual Learning (2021, background): “We study the common continual learning paradigm in which pre-training precedes continual learning [28, 29, 55, 10, 71, 32, 33, 34, 68, 4].”
Semantic-aware Knowledge Distillation for Few-Shot Class-Incremental Learning (2021, methodology): “[40] introduced an approach to train an individual network for the novel classes, and then merging this new network with the network based on previous classes using a double distillation objective.”
Online Continual Learning in Image Classification: An Empirical Survey (2021, methodology): “DMC [73] trains a separate model for new tasks and combines the new and old models using”
SID: Incremental Learning for Anchor-Free Object Detection via Selective and Inter-Related Distillation (2020, methodology): “We follow the widely used protocol in current literature (Shmelkov et al., 2017; Zhang et al., 2020) — the new categories are continually added in alphabetical order.”
Incremental Learning via Rate Reduction (2020, background): “Recent works [22, 25] explore the more difﬁcult class incremental (class-IL) setting, where task-ID is withheld at inference time.”
Memory-Efficient Incremental Learning Through Feature Adaptation (2020, background): “[56] argue that LwF produces models that are either biased towards old or new classes.”