A Model or 603 Exemplars: Towards Memory-Efficient Class-Incremental Learning

By Da-Wei Zhou, Qiwen Wang, Han-Jia Ye, De-chuan Zhan. 2022

241 citations recorded by the source corpus.

Real-world applications require the classification model to adapt to new classes without forgetting old ones. Correspondingly, Class-Incremental Learning (CIL) aims to train a model with limited memory size to meet this requirement. Typical CIL methods tend to save representative exemplars from former classes to resist forgetting, while recent works find that storing models from history can substantially boost the performance. However, the stored models are not counted into the memory budget, which implicitly results in unfair comparisons. We find that when counting the model size into the total budget and comparing methods with aligned memory size, saving models do not consistently work, especially for the case with limited memory budgets. As a result, we need to holistically evaluate different CIL methods at different memory scales and simultaneously consider accuracy and memory size for measurement. On the other hand, we dive deeply into the construction of the memory buffer for memory efficiency. By analyzing the effect of different layers in the network, we find that shallow and deep layers have different characteristics in CIL. Motivated by this, we propose a simple yet effective baseline, denoted as MEMO for Memory-efficient Expandable MOdel. MEMO extends specialized layers based on the shared generalized representations, efficiently extracting diverse representations with modest cost and maintaining representative exemplars. Extensive experiments on benchmark datasets validate MEMO's competitive performance. Code is available at: https://github.com/wangkiw/ICLR23-MEMO

Subfields: Artificial Intelligence, Health Information Management

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Health Information Management

Citations in the corpus

Few-Shot Class-Incremental Learning by Sampling Multi-Phase Tasks (2022, methodology): “Other non-exemplar-based methods change the network structure by expanding and pruning to meet the requirements of new tasks [31], [32], [33], [34].”
Expandable Subspace Ensemble for Pre-Trained Model-Based Class-Incremental Learning (2024, methodology): “• MEMO [77] : aims to alleviate the memory cost of DER from another aspect.”
Class-Incremental Learning: A Survey (2023, methodology): “It must be noted that Finetune, EWC, LwF, and L2P are exemplar-free methods, and we do not use any exemplar set for them.”
Deep Class-Incremental Learning: A Survey (2023, methodology): “Learning dynamic networks, especially backbone expansion methods, have achieved state-of-the-art performance in recent years [22].”
Online Continual Learning For Interactive Instruction Following Agents (2024, background): “Memory requirements make them unsuitable for settings with limited memory in edge devices (Zhou et al., 2022a).”
Elastic Feature Consolidation for Cold Start Exemplar-free Incremental Learning (2024, background): “Yan et al. (2021) train a single backbone for each incremental task, and more recently Zhou et al. (2022) share generalized residual blocks and extend specialized blocks for new tasks to improve performance and efficiency.”
Continual Segmentation with Disentangled Objectness Learning and Class Recognition (2024, unknown): “MEMO [64] went further by decoupling the network and only expanding specialized layers.”