Learning Without Memorizing

By Prithviraj Dhar, Rajat Vikram Singh, Kuan-Chuan Peng, Ziyan Wu, Ramalingam Chellappa. 2018

585 citations recorded by the source corpus.

Incremental learning (IL) is an important task aimed at increasing the capability of a trained model, in terms of the number of classes recognizable by the model. The key problem in this task is the requirement of storing data (e.g. images) associated with existing classes, while teaching the classifier to learn new classes. However, this is impractical as it increases the memory requirement at every incremental step, which makes it impossible to implement IL algorithms on edge devices with limited memory. Hence, we propose a novel approach, called `Learning without Memorizing (LwM)', to preserve the information about existing (base) classes, without storing any of their data, while making the classifier progressively learn the new classes. In LwM, we present an information preserving penalty: Attention Distillation Loss (L_{AD}), and demonstrate that penalizing the changes in classifiers' attention maps helps to retain information of the base classes, as new classes are added. We show that adding L_{AD} to the distillation loss which is an existing information preserving loss consistently outperforms the state-of-the-art performance in the iILSVRC-small and iCIFAR-100 datasets in terms of the overall accuracy of base and incrementally learned classes.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical Learning Theory

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “LwM [119] exploits the attention maps of new training samples for KD. EBLL [121] learns task-speciﬁc autoencoders and prevents changes in feature reconstruction.”
Learnability and Algorithm for Continual Learning (2023, methodology): “Many existing approaches have used similar approaches (Ritter et al., 2018; Schwarz et al., 2018; Xu & Zhu, 2018; Castro et al., 2018; Dhar et al., 2019; Lee et al., 2019; Ahn et al., 2019; Liu et al., 2020a).”
SLCA: Slow Learner with Classifier Alignment for Continual Learning on a Pre-trained Model (2023, background): “Representative strategies include regularization-based approaches [19, 1, 51, 25, 41, 7], which preserve the old model and selectively stabilize changes of parameters or predictions; replaybased approaches [49, 30, 3, 44, 40, 39], which approximate and recover the previously-learned data distributio”
A Theoretical Study on Solving Continual Learning (2022, methodology): “Using regularization [12, 13] to minimize changes to model parameters learned from previous tasks is a popular approach [14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26].”
DLCFT: Deep Linear Continual Fine-Tuning for General Incremental Learning (2022, methodology): “Regularization-based methods aim to penalize the update either in function space [8, 24] or in parameter space [2, 9, 18, 20, 22, 23, 32, 38, 45].”
Class-Incremental Learning with Cross-Space Clustering and Controlled Transfer (2022, methodology): “Early works in this category distill logit scores [32, 42] or attention maps [12] of the previous model.”
Class-incremental Novel Class Discovery (2022, methodology): “Existing IL methods can be sub-divided into three broad categories: regularization-based methods [22,34,27,9], exemplarbased methods [31,5,2,7] and methods focused on task-recency bias problem [32].”
Adaptive Orthogonal Projection for Batch and Online Continual Learning (2022, background): “Many other papers use related approaches (Zenke, Poole, and Ganguli 2017; Fernando et al. 2017; Aljundi et al. 2017; Ritter, Botev, and Barber 2018; Xu and Zhu 2018; Kemker and Kanan 2018; Parisi et al. 2018; Ahn et al. 2019; Hu et al. 2021; Dhar et al. 2019; Adel, Zhao, and Turner 2020).”
Class-Incremental Learning by Knowledge Distillation with Adaptive Feature Consolidation (2022, background): “On the other hand, Learning without Memorizing (LwM) [11] minimizes the difference of the gradients given by the highest score classes in the old tasks with respect to intermediate feature maps, which is motivated by [42].”
Uncertainty-aware Contrastive Distillation for Incremental Semantic Segmentation (2022, background): “[38] proposed to penalize the changes in attention maps.”
Representation Compensation Networks for Continual Semantic Segmentation (2022, neutral): “In the more challenging scenario without any replay, many methods explore regularization to maintain old knowledge, including knowledge distillation [10, 18, 21, 28, 51, 69, 74, 100], adversarial training [29,88], vanilla regularization [43,48,56,66,67,80,95,98] and so on.”
TRGP: Trust Region Gradient Projection for Continual Learning (2022, methodology): “This category of methods (e.g., (Kirkpatrick et al., 2017; Lee et al., 2017; Chaudhry et al., 2018a; Dhar et al., 2019; Ritter et al., 2018; Schwarz et al., 2018; Zenke et al., 2017)) protect the old tasks by adding regularization terms in the loss function to penalize the model change on their…”
vCLIMB: A Novel Video Class Incremental Learning Benchmark (2022, methodology): “Continual learning [11] directly models such a scenario, by adapting a neural network model into a sequential series of tasks.”
Continual Learning of a Mixed Sequence of Similar and Dissimilar Tasks (2021, neutral): “Dhar et al. (2019) combined three loss functions to encourage the model resulted from the new task to be similar to the previous model.”
Incremental Learning in Semantic Segmentation from Image Labels (2021, background): “Even if in image classification it has been exhaustively studied [1, 11, 20, 22, 24, 30, 38, 41, 54, 57, 68], in semantic segmentation it is still in its early stages [8, 9, 21, 33, 43, 45–47].”
DyTox: Transformers for Continual Learning with DYnamic TOken eXpansion (2021, background): “These constraints can be directly applied on the weights [39, 79, 1, 8], intermediary features [32, 15, 83, 19, 17], prediction probabilities [47, 59, 5, 6], or on the gradients [51, 9, 21, 62].”
Overcoming Catastrophic Forgetting in Incremental Few-Shot Learning by Finding Flat Minima (2021, background): “Incremental learning focuses on the challenging problem of continually learning to recognize new classes in new coming data without forgetting old classes [6, 7, 10, 51].”
Bridging Non Co-occurrence with Unlabeled In-the-wild Data for Incremental Object Detection (2021, neutral): “The second type of regularization-based approaches is based on knowledge distillation [14, 6].”