DeeSIL: Deep-Shallow Incremental Learning

By Eden Belouadah, Adrian Daniel Popescu. 2018

77 citations recorded by the source corpus.

Incremental Learning (IL) is an interesting AI problem when the algorithm is assumed to work on a budget. This is especially true when IL is modeled using a deep learning approach, where two complex challenges arise due to limited memory, which induces catastrophic forgetting and delays related to the retraining needed in order to incorporate new classes. Here we introduce DeeSIL, an adaptation of a known transfer learning scheme that combines a fixed deep representation used as feature extractor and learning independent shallow classifiers to increase recognition capacity. This scheme tackles the two aforementioned challenges since it works well with a limited memory budget and each new concept can be added within a minute. Moreover, since no deep retraining is needed when the model is incremented, DeeSIL can integrate larger amounts of initial data that provide more transferable features. Performance is evaluated on ImageNet LSVRC 2012 against three state of the art algorithms. Results show that, at scale, DeeSIL performance is 23 and 33 points higher than the best baseline when using the same and more initial data respectively.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

In Meta & Continual Learning

Citations in the corpus

FeTrIL: Feature Translation for Exemplar-Free Class-Incremental Learning (2022, methodology): “The nearest class mean (NCM) [26] was used in [33], linear SVMs [30] were used in [1] and extreme value machines [34] were recently tested by [6].”
Self-Supervised Training Enhances Online Continual Learning (2021, methodology): “We study the common continual learning paradigm in which pre-training precedes continual learning [28, 29, 55, 10, 71, 32, 33, 34, 68, 4].”
MgSvF: Multi-Grained Slow versus Fast Framework for Few-Shot Class-Incremental Learning (2020, background): “Incremental learning is usually conducted under the task-incremental [72], [73] or the class-incre-mental learning scenarios [2], [70], [74], [75], [76], [77].”
ScaIL: Classifier Weights Scaling for Class Incremental Learning (2020, methodology): “DeeSIL [3] is an adaptation of transfer learning to a class IL context [13, 15].”
REMIND Your Neural Network to Prevent Catastrophic Forgetting (2019, methodology): “t scale to datasets with thousands of categories, making them ill-suited for robotics applications. While there are incremental batch learning methods that do not use test-time task labels [5], [15], [27], these methods operate on vectors, rather than tensors and they cannot be easily used to train ”
IL2M: Class Incremental Learning With Dual Memory (2019, methodology): “The comparison of FT to DeeSIL [2] is also favorable for all settings where K > 0, except for Z = 5 and ILSVRC with Z = 10 and K = {5000, 10000}.”
A Comprehensive Study of Class Incremental Learning Algorithms for Visual Tasks (2020, methodology): “Yet another choice [4] is to exploit a ﬁxed deep representation to extract features for all incremental states and a battery of linear SVMs to implement C .”
Regularization Shortcomings for Continual Learning (2019, background): “• Rehearsal: In order to maintain knowledge from past learning experiences, the algorithms save a subset of old data as memory [21, 19, 1, 2, 8, 28, 7, 3, 26].”
Incremental Learning with Maximum Entropy Regularization: Rethinking Forgetting and Intransigence (2019, background): “Lee et al. and Belouadah et al. propose a network architecture whose capacity is dynamically expandable (Lee et al., 2017; Belouadah & Popescu, 2018).”
IExpressNet: Facial Expression Recognition with Incremental Classes (2020, background): “The second kind of methods [4, 28] try to learn a fix representation in the initial state for the whole following process.”
Understanding Continual Learning Settings with Data Distribution Drift Analysis (2021, background): “Indeed, in a wide part of continual learning literature, incremental learning is associated with the “disjoint class scenarios” ( virtual concept drifts ) (Belouadah and Popescu, 2018; Wu et al., 2019; Douillard et al., 2020b).”
Continual Learning with Foundation Models: An Empirical Study of Latent Replay (2022, background): “…and applicable in large set of scenarios (Kalifou et al., 2019; Diethe et al., 2018; Lesort et al., 2019; Prabhu et al., 2020; Traoré et al., 2019; Belouadah and Popescu, 2018; Wu et al., 2019; Hou et al., 2019; Lesort, 2020; Douillard et al., 2020a) in comparison with other families of CL…”
Brain-Inspired Model for Incremental Learning Using a Few Examples (2020, methodology): “Furthermore, compared to methods that store only 2000 images for previous classes [35, 6, 4], 7500 centroids for our approach require less memory than 2000 complete images.”
I3DOL: Incremental 3D Object Learning without Catastrophic Forgetting (2020, background): “When a small number of exemplars from each old class are selected for training, (Rebuffi et al. 2017; Castro et al. 2018; Belouadah and Popescu 2018) focus on alleviating the effect of unbalanced training samples between the past and new categories.”
Continual Learning in Deep Networks: an Analysis of the Last Layer (2021, background): “Continual learning research field studied many strategies, such as rehearsal [26, 9, 2, 3, 52, 19, 6], generative replay [46, 53, 28, 30, 40], dynamic architecture [16, 42, 32] or regularization [22, 55, 41, 44].”
Elastic Feature Consolidation for Cold Start Exemplar-free Incremental Learning (2024, methodology): “Most functional regularization approaches use feature distillation (Zhu et al., 2021a; Yu et al., 2020), or even freeze the feature extractor after the first task (Petit et al., 2023; Belouadah & Popescu, 2018; Panos et al., 2023), greatly reducing the plasticity of the network.”
Queried Unlabeled Data Improves and Robustifies Class-Incremental Learning (2022, unknown): “Another category of solutions is based on transfer learning (Kemker & Kanan, 2018; Belouadah & Popescu, 2018), whose eﬀectiveness yet hinges on the pre-training quality.”
Self-Supervised Features Improve Open-World Learning (2021, background): “conducted a comprehensive evaluation of recently proposed incremental learning approaches [37, 52, 29, 58, 3, 4, 2, 8, 24, 25] and concluded that none of the existing algorithms was better than the others; both memory and incremental step size influence the relative performance.”