Revisiting Distillation and Incremental Classifier Learning

By Khurram Javed, Faisal Shafait. 2018

69 citations recorded by the source corpus.

One of the key differences between the learning mechanism of humans and Artificial Neural Networks (ANNs) is the ability of humans to learn one task at a time. ANNs, on the other hand, can only learn multiple tasks simultaneously. Any attempts at learning new tasks incrementally cause them to completely forget about previous tasks. This lack of ability to learn incrementally, called Catastrophic Forgetting, is considered a major hurdle in building a true AI system. In this paper, our goal is to isolate the truly effective existing ideas for incremental learning from those that only work under certain conditions. To this end, we first thoroughly analyze the current state of the art (iCaRL) method for incremental learning and demonstrate that the good performance of the system is not because of the reasons presented in the existing literature. We conclude that the success of iCaRL is primarily due to knowledge distillation and recognize a key limitation of knowledge distillation, i.e, it often leads to bias in classifiers. Finally, we propose a dynamic threshold moving algorithm that is able to successfully remove this bias. We demonstrate the effectiveness of our algorithm on CIFAR100 and MNIST datasets showing near-optimal results. Our implementation is available at this https URL.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

In Meta & Continual Learning

Citations in the corpus

Adaptive Orthogonal Projection for Batch and Online Continual Learning (2022, methodology): “In testing, a distance function over the nearest exemplar/class mean is used for classiﬁcation (Rebufﬁ et al. 2017; Lee et al. 2018; Javed and Shafait 2018; Bendale and Boult 2015).”
Continual Normalization: Rethinking Batch Normalization for Online Continual Learning (2022, background): “However, empirically they provide insigniﬁcant improvements over the random sampling strategies (Javed & Shafait, 2018; Chaudhry et al., 2019a).”
Continual Learning by Using Information of Each Class Holistically (2021, methodology): “In testing, a distance function over nearest exemplar/class mean is used for classiﬁcation (Re-bufﬁ, Kolesnikov, and Lampert 2017; Lee et al. 2018; Javed and Shafait 2018; Bendale and Boult 2015).”
Online Continual Learning in Image Classification: An Empirical Survey (2021, background): “Multiple recent works have proposed different ways to combine the KD loss with the classification loss [7, 40, 101].”
Memory-Efficient Incremental Learning Through Feature Adaptation (2020, background): “Similarly, Javed and Shafait [25] learn an end-to-end classiﬁer by proposing a dynamic threshold moving algorithm.”
SS-IL: Separated Softmax for Incremental Learning (2020, methodology): “[18] points out that methods using exemplar-memory cause imbalanced dataset and consequently have been shown to suffer from the bias problem in the final FC layer.”
IL2M: Class Incremental Learning With Dual Memory (2019, methodology): “A ResNet-18 architecture [10] was used in [24] and then in [12] and [5].”
Overcoming Catastrophic Forgetting With Unlabeled Data in the Wild (2019, methodology): “On the other hand, data-based approaches [3, 12, 13, 24, 33] keep the knowledge of the previous tasks by knowledge distillation [11], which minimizes the distance between the manifold of the latent space in the previous and new models.”
Class-incremental Learning via Deep Model Consolidation (2019, methodology): “Although some recent works [42, 19, 3, 15, 35, 6] that rely on the storage of partial old data have made impressive progress, this direction is arguably not memory efficient and it may violate some practical constraints such as copyright or privacy issues.”
Class-Incremental Learning: Survey and Performance Evaluation on Image Classification (2020, methodology): “Based on an analysis of iCaRL, Javed and Shafait [52] propose a dynamic threshold moving algorithm to fix the nearest exemplar mean classifier bias trained with distillation by maintaining an up-to-date scaling vector.”
Lifelong Machine Learning with Deep Streaming Linear Discriminant Analysis (2019, methodology): “In conjunction with storing and replaying previous samples, many methods use a distillation loss [25] to regularize weight updates so that the network does not drift far from its previous solu-tion [9, 26, 29, 44, 50].”
M2KD: Multi-model and Multi-level Knowledge Distillation for Incremental Learning (2019, methodology): “[16] alleviate the bias in knowledge distillation by introducing a scaling vector to trained classifier, however, our approach is agnostic to classifier and achieves better”
A Wholistic View of Continual Learning with Deep Neural Networks: Forgotten Lessons and the Bridge to Active and Open World Learning (2020, background): “The other two (De Lange et al., 2021; Javed and Shafait, 2018) follow this practice, but go even further and make the statement that task ordering has minimal inﬂuence towards continual learning methods.”
A Comprehensive Study of Class Incremental Learning Algorithms for Visual Tasks (2020, methodology): “Different ﬂavors of distillation were later proposed in [10, 19, 20, 58].”
Incremental learning for the detection and classification of GAN-generated images (2019, background): “We also compare our proposal with the basic iCaRL version (that is, Single Task and Single Classifier) and two more strategies proposed in [20], referred to as S-Classifier and GS-Classifier.”
Learning a Unified Classifier Incrementally via Rebalancing (2019, methodology): “[4] apply threshold moving [1] (denoted as iCaRL-TM) to the CNN prediction of iCaRL, while the performance is still a little inferior to iCaRL-NME.”
Overcoming Catastrophic Forgetting by Bayesian Generative Regularization (2019, methodology): “…instead of ﬁxing a model, memory-based methods store a subset of previous data and constrain the update of models by leveraging the distilled knowledge from previous tasks (Castro et al., 2018; Hou et al., 2018; Javed & Shafait, 2018; Li & Hoiem, 2017; Rebufﬁ et al., 2017; Shin et al., 2017).”
Long Live the Lottery: The Existence of Winning Tickets in Lifelong Learning (2021, background): “Besides, several approaches are learning with memorized data (Castro et al., 2018; Javed & Shafait, 2018; Rebuffi et al., 2017; Belouadah & Popescu, 2019; 2020; Lopez-Paz & Ranzato, 2017; Chaudhry et al., 2018).”