Initial Classifier Weights Replay for Memoryless Class Incremental Learning

By Eden Belouadah, Adrian Daniel Popescu, Ioannis Kanellos. 2020

24 citations recorded by the source corpus.

Incremental Learning (IL) is useful when artificial systems need to deal with streams of data and do not have access to all data at all times. The most challenging setting requires a constant complexity of the deep model and an incremental model update without access to a bounded memory of past data. Then, the representations of past classes are strongly affected by catastrophic forgetting. To mitigate its negative effect, an adapted fine tuning which includes knowledge distillation is usually deployed. We propose a different approach based on a vanilla fine tuning backbone. It leverages initial classifier weights which provide a strong representation of past classes because they are trained with all class data. However, the magnitude of classifiers learned in different states varies and normalization is needed for a fair handling of all classes. Normalization is performed by standardizing the initial classifier weights, which are assumed to be normally distributed. In addition, a calibration of prediction scores is done by using state level statistics to further improve classification fairness. We conduct a thorough evaluation with four public datasets in a memoryless incremental learning setting. Results show that our method outperforms existing techniques by a large margin for large-scale datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

Recent Advances of Continual Learning in Computer Vision: An Overview (2021, neutral): “Another similar method [94] was also proposed to standardize the stored initial classifier of each task, resulting in a fair and balanced classification among different tasks.”
Continual Contrastive Self-supervised Learning for Image Classification (2021, background): “Continual learning is also referred to as lifelong learning[13][1][9][37], sequential learning[4][43] or incremental learning[2][8][40][5].”
Energy-Based Models for Continual Learning (2020, background): “Another important distinction in CL is between taskincremental learning (Task-IL) and class-incremental learning (Class-IL) (van de Ven & Tolias, 2019; Prabhu et al., 2020; Belouadah et al., 2020; Maltoni & Lomonaco, 2019; Hayes & Kanan, 2020).”
Class-Incremental Learning with Generative Classifiers (2021, background): “safety or privacy concerns), and in this study we do not allow data storage, a setting which has been referred to as memoryless class-incremental learning [7].”