Small-Task Incremental Learning
By Arthur Douillard, Matthieu Cord, Charles Ollion, Thomas Robert, Eduardo Valle. 2020
7 citations recorded by the source corpus.
Lifelong learning has attracted much attention, but existing works still struggle to fight catastrophic forgetting and accumulate knowledge over long stretches of incremental learning. In this work, we propose PODNet, a model inspired by representation learning. By carefully balancing the compromise between remembering the old classes and learning new ones, PODNet fights catastrophic forgetting, even over very long runs of small incremental tasks-- a setting so far unexplored by current works. PODNet innovates on existing art with an efficient spatial-based distillation-loss applied throughout the model and a representation comprising multiple proxy vectors for each class. We validate those innovations thoroughly, comparing PODNet with three state-of-the-art models on three datasets: CIFAR100, ImageNet100, and ImageNet1000. Our results showcase a significant advantage of PODNet over existing art, with accuracy gains of 12.10, 4.83, and 2.85 percentage points, respectively. Code will be released at this address: https://github.com/arthurdouillard/incremental_learning.pytorch.
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
- Arthur Douillard
- Hanbin Zhao
- Da-Wei Zhou
- Ameya Prabhu
- Yaoyao Liu
- Matthieu Cord
- Philip H. S. Torr
- Bernt Schiele
Citations in the corpus
- MgSvF: Multi-Grained Slow versus Fast Framework for Few-Shot Class-Incremental Learning (2020, methodology): “We use an embedding network as the feature extractor with the feature dimension being 512.”
- GDumb: A Simple Approach that Questions Our Progress in Continual Learning (2020, methodology): “Regularize Memory Distill Param iso × × × BiC[16], iCARL[4] × × UCIR[17], EEIL[18] IL2M[19], WA[20] PODNet[21], MCIL[22] RPS-Net[23], iTAML[24] × CGATE[25] × × RWALK[8] × × Here we categorize various recently proposed CL approaches depending on the underlying simplifying assumptions they impose. of ”
- Insights from the Future for Continual Learning (2020, methodology): “The results appear in Table 1, which shows, for the datasets and protocols explained in the top of this section, the performance for our two base models [17, 7], and the improvements on those base models as we implement our proposed model, with and without the SVM latent-space regularization refinem”