DER: Dynamically Expandable Representation for Class Incremental Learning

By Shipeng Yan, Jiangwei Xie, Xuming He. 2021

671 citations recorded by the source corpus.

We address the problem of class incremental learning, which is a core step towards achieving adaptive vision intelligence. In particular, we consider the task setting of incremental learning with limited memory and aim to achieve better stability-plasticity trade-off. To this end, we propose a novel two-stage learning approach that utilizes a dynamically expandable representation for more effective incremental concept modeling. Specifically, at each incremental step, we freeze the previously learned representation and augment it with additional feature dimensions from a new learnable feature extractor. This enables us to integrate new visual concepts with retaining learned knowledge. We dynamically expand the representation according to the complexity of novel concepts by introducing a channel-level mask-based pruning strategy. Moreover, we introduce an auxiliary loss to encourage the model to learn diverse and discriminate features for novel concepts. We conduct extensive experiments on the three class incremental learning benchmarks and our method consistently outperforms other methods with a large margin.1

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, methodology): “ORDER [220] employs unlabeled OOD data with experience replay and feature replay to cope with highly dynamic task distributions.”
Learnability and Algorithm for Continual Learning (2023, neutral): “…buffer data (Rusu et al., 2016; LopezPaz & Ranzato, 2017; Chaudhry et al., 2019a; Hou et al., 2019; Wu et al., 2019; Rolnick et al., 2019; Buzzega et al., 2020; Rajasegaran et al., 2020a; Liu et al., 2021; Cha et al., 2021; Yan et al., 2021; Wang et al., 2022b; Guo et al., 2022; Kim et al., 2022a).”
The Ideal Continual Learner: An Agent That Never Forgets (2023, methodology): “, 2022), or combinations thereof (Yan et al., 2021; Xie et al., 2022; Wang et al., 2022b; Frascaroli et al., 2023); for surveys, see, e.”
First Session Adaptation: A Strong Replay-Free Baseline for Class-Incremental Learning (2023, methodology): “We show that this approach is comparable to a softmax classifier in the offline setting and that it outperforms NCM. Due to the challenging nature of class incremental learning, some methods employ a buffer of stored samples from previous sessions [24, 39].”
A Theoretical Study on Solving Continual Learning (2022, background): “DER can generate a large number of parameters after the last task, e.g., 117.6 millions (M) for C100-20T while our proposed methods require 44.6M (HAT+CSI) and 11.6M (Sup+CSI) (refer to Appendix H).”
Generative Negative Text Replay for Continual Vision-Language Pretraining (2022, methodology): “The third is the structure methods [49,39], which keep old parameters fixed while growing and allocating weights for learning new data.”
Continual Learning by Modeling Intra-Class Variation (2022, methodology): “, 2020), dynamic architecture-based methods (Rajasegaran et al., 2019; Hung et al., 2019; Yan et al., 2021) and memory-based methods (Rebuffi et al.”
Long-Tailed Class Incremental Learning (2022, methodology): “Further results on AANets [23] and DER [43] We report results for AANets (based on LUCIR) on Shuffle LT-CIL scenario (CIFAR-100 with 10- task setting).”
Class-Incremental Learning with Cross-Space Clustering and Controlled Transfer (2022, methodology): “Architecture-based methods modify the network architecture dynamically to fit more tasks, by expanding the model by adding more weights [55, 52], or masking and allocating subnetworks to specific tasks [45], or by gating the parameters dynamically using a task identifier [1].”
Online Continual Learning with Contrastive Vision Transformer (2022, methodology): “Following the setting proposed in [62,89], we trains all 100 classes in several splits, including 10, 20 incremental tasks.”
CoSCL: Cooperation of Small Continual Learners is Stronger than a Big One (2022, neutral): “Representative strategies include weight regularization [20,52,1], memory replay [36,44,46], parameter isolation [41,18] and dynamic architecture [14,50].”
Learning to Imagine: Diversify Memory for Incremental Learning using Unlabeled Data (2022, methodology): “DER [33] expands the feature extracting backbone at each training step and tries to reduce parameters via pruning by a learnable channel-level mask.”
FOSTER: Feature Boosting and Compression for Class-Incremental Learning (2022, methodology): “Besides, even with restrictions of KD, the model still suffer from feature degradation [40] of old concepts due to limited access [5] to old data.”
Class-Incremental Learning by Knowledge Distillation with Adaptive Feature Consolidation (2022, methodology): “Architectural approaches employ network expansion schemes [27, 41, 49] instead of using static models.”
Few-Shot Class-Incremental Learning by Sampling Multi-Phase Tasks (2022, methodology): “Other non-exemplar-based methods change the network structure by expanding and pruning to meet the requirements of new tasks [31], [32], [33], [34].”