Adaptive Deep Models for Incremental Learning: Considering Capacity Scalability and Sustainability

By Yang Yang, Da-Wei Zhou, De-chuan Zhan, Hui Xiong, Yuan Jiang. 2019

112 citations recorded by the source corpus.

Recent years have witnessed growing interests in developing deep models for incremental learning. However, existing approaches often utilize the fixed structure and online backpropagation for deep model optimization, which is difficult to be implemented for incremental data scenarios. Indeed, for streaming data, there are two main challenges for building deep incremental models. First, there is a requirement to develop deep incremental models with Capacity Scalability. In other words, the entire training data are not available before learning the task. It is a challenge to make the deep model structure scaling with streaming data for flexible model evolution and faster convergence. Second, since the stream data distribution usually changes in nature (concept drift), there is a constraint for Capacity Sustainability. That is, how to update the model while preserving previous knowledge for overcoming the catastrophic forgetting. To this end, in this paper, we develop an incremental adaptive deep model (IADM) for dealing with the above two capacity challenges in real-world incremental data scenarios. Specifically, IADM provides an extra attention model for the hidden layers, which aims to learn deep models with adaptive depth from streaming data and enables capacity scalability. Also, we address capacity sustainability by exploiting the attention based fisher information matrix, which can prevent the forgetting in consequence. Finally, we conduct extensive experiments on real-world data and show that IADM outperforms the state-of-the-art methods with a substantial margin. Moreover, we show that IADM has better capacity scalability and sustainability in incremental learning scenarios.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Han-Jia Ye
2. De-chuan Zhan
3. Da-Wei Zhou
4. Qiwen Wang
5. Ziwei Liu
6. Zhiyuan Qi
7. Zhi-Hong Qi
8. Latifur R. Khan

In Electrical and Electronic Engineering

1. Han-Jia Ye
2. De-chuan Zhan
3. Da-Wei Zhou
4. Ziwei Liu

In Management Science and Operations Research

1. Han-Jia Ye
2. De-chuan Zhan
3. Da-Wei Zhou
4. Ziwei Liu
5. Xujiang Zhao

Citations in the corpus

• Co-Transport for Class-Incremental Learning (2021, background): “, e-commerce product search [43], video action recognition[54], natural language generation [30], multimedia retrieval [40], and social media topic mining [49].”
• Learning Placeholders for Open-Set Recognition (2021, background): “Since it is impossible to cover all classes in the world as training set [19, 33], the model would treat all novel category instances as known ones.”
• Class-Incremental Learning: A Survey (2023, methodology): “For example, Adam-NSCL [55] proposes to sequentially optimize network parameters by projecting the candidate parameter update into the approximate null space of all previous tasks.”
• Learning to Classify With Incremental New Class (2021, unknown): “LACU-SVM requires additional unlabeled instances in training and every model update, and ECSMiner needs all the true labels to train a new classifier for the ensemble.”
• Cost-Effective Incremental Deep Model: Matching Model Capacity With the Least Sampling (2023, unknown): “Then, with the newly labeled examples and existing coming labeled data, we incrementally train the deep model (e.g., the IDM approach [45] in the conference version) considering both the scalability and sustainability.”
• Mitigating Catastrophic Forgetting in Online Continual Learning by Modeling Previous Task Interrelations via Pareto Optimization (2024, background): “Specifically, regularization-based meth-ods (Kirkpatrick et al., 2017; Husz´ar, 2017; Ritter et al., 2018; Zenke et al., 2017; Yang et al., 2019) intend to design different regularization techniques to preserve important parameters for previously learned tasks.”
• CLEAR: Contrastive-Prototype Learning with Drift Estimation for Resource Constrained Stream Mining (2021, background): “This setting, also known as continual/incremental learning [32, 41], has attracted remarkable attention, featured by the the challenges of Concept Evolution [25, 30, 58] and Catastrophic Forgetting [20, 26, 50, 53].”
• Deep Class-Incremental Learning: A Survey (2023, methodology): “Hence, IADM augments EWC with an ensemble of different layers and learns layer-wise importance matrix in an online manner.”