Memory-Efficient Incremental Learning Through Feature Adaptation

By Ahmet Iscen, Jeffrey O. Zhang, Svetlana Lazebnik, Cordelia Schmid. 2020

207 citations recorded by the source corpus.

We introduce an approach for incremental learning that preserves feature descriptors of training images from previously learned classes, instead of the images themselves, unlike most existing work. Keeping the much lower-dimensional feature embeddings of images reduces the memory footprint significantly. We assume that the model is updated incrementally for new classes as new data becomes available sequentially.This requires adapting the previously stored feature vectors to the updated feature space without having access to the corresponding original training images. Feature adaptation is learned with a multi-layer perceptron, which is trained on feature pairs corresponding to the outputs of the original and updated network on a training image. We validate experimentally that such a transformation generalizes well to the features of the previous set of classes, and maps features to a discriminative subspace in the feature space. As a result, the classifier is optimized jointly over new and old classes without requiring old class images. Experimental results show that our method achieves state-of-the-art classification accuracy in incremental learning benchmarks, while having at least an order of magnitude lower memory footprint compared to image-preserving strategies.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, methodology): “The alternatives can be new training samples [92], [180], [252], [354], a small fraction of old training samples [53], [100], [164], [356], external unlabeled data [239], generated data [452], [489], etc.”
RMM: Reinforced Memory Management for Class-Incremental Learning (2023, methodology): “Reinforcement Learning deﬁnes an agent that needs to decide its actions in an unknown environment by maximizing the expected cumulative reward.”
ALIFE: Adaptive Logit Regularizer and Feature Replay for Incremental Semantic Segmentation (2022, methodology): “Table 4: Quantitative results on PASCAL VOC [10] in terms of IoU scores.”
FOSTER: Feature Boosting and Compression for Class-Incremental Learning (2022, background): “[21] preserves low dimensional features instead of raw instances to reduce the storage overhead.”
Few-Shot Class-Incremental Learning by Sampling Multi-Phase Tasks (2022, background): “Other works consider saving embeddings instead of raw images [39], using generative models for data rehearsal [40], taskwise adaptation [41], and output normalization [20], [42], [43].”
Uncertainty-aware Contrastive Distillation for Incremental Semantic Segmentation (2022, methodology): “Following the previous works [23], [29], [66] the comparison is performed under three different settings differentiating mainly in how the new task data is handled: 100 classes in the old task and 50 classes at once for the new one (100-50), same number of old classes and 50 classes in batches of 10”
Representation Compensation Networks for Continual Semantic Segmentation (2022, methodology): “Some methods [43,48,56,66,67,80,95] play regularization on model parameters to improve its stability.”
Self-Supervised Models are Continual Learners (2021, methodology): “[17] is also related to CaSSLe, but it focuses on memory efficiency which is less interesting in our setting.”
DyTox: Transformers for Continual Learning with DYnamic TOken eXpansion (2021, methodology): “In Table 3, we compare the impact of sharing the TAB per task — and only maintain different tokens per task.”
Recent Advances of Continual Learning in Computer Vision: An Overview (2021, neutral): “Besides topology information, a few other meth-ods [120, 121] proposed to preserve various other information.”
Co-Transport for Class-Incremental Learning (2021, background): “[16] proposes to select features instead of images, and reduces the storage cost.”
Optimizing Reusable Knowledge for Continual Learning via Metalearning (2021, background): “Similarly, [36, 37] seek to be memory-efficient by saving feature vectors of instances from previous tasks, while learning a transformation from the feature space of past tasks to current ones.”
On Learning the Geodesic Path for Incremental Learning (2021, neutral): “[18] proposes a feature adaptation strategy.”
Replay in Deep Learning: Current Approaches and Missing Biological Elements (2021, positive): “For example, several methods using representational replay (Hayes et al., 2020; Caccia et al., 2020; Pellegrini et al., 2019; Iscen et al., 2020) or generative representational replay (van de Ven et al., 2020; Lao et al., 2020), instead of veridical (raw pixel) replay, have been proposed to improve…”
Rainbow Memory: Continual Learning with a Memory of Diverse Samples (2021, background): “Iscen et al . propose to reduce the dimension of stored features for efﬁciency [22].”
Online Continual Learning in Image Classification: An Empirical Survey (2021, background): “Feature replay is a promising direction where latent features of the old samples at a given layer (feature extraction layer) are relayed instead of raw data [64, 115, 116, 117].”
PLOP: Learning without Forgetting for Continual Semantic Segmentation (2020, background): “Several solutions exist to address this problem: for instance, rehearsal learning consists in keeping a limited amount of training data from old classes either as raw images [55, 54, 6, 10], compressed features [24, 33], or generated training data [34, 60, 43].”
Lifelong Graph Learning (2020, neutral): “To reduce the memory consumption of exemplars, [18] applied the distillation loss to feature space without having to access to the corresponding images.”