On Learning the Geodesic Path for Incremental Learning

By Christian Simon, Piotr Koniusz, Mehrtash Harandi. 2021

155 citations recorded by the source corpus.

Neural networks notoriously suffer from the problem of catastrophic forgetting, the phenomenon of forgetting the past knowledge when acquiring new knowledge. Overcoming catastrophic forgetting is of significant importance to emulate the process of "incremental learning", where the model is capable of learning from sequential experience in an efficient and robust way. State-of-the-art techniques for incremental learning make use of knowledge distillation towards preventing catastrophic forgetting. Therein, one updates the network while ensuring that the network’s responses to previously seen concepts remain stable throughout updates. This in practice is done by minimizing the dissimilarity between current and previous responses of the network one way or another. Our work contributes a novel method to the arsenal of distillation techniques. In contrast to the previous state of the art, we propose to firstly construct low-dimensional manifolds for previous and current responses and minimize the dissimilarity between the responses along the geodesic connecting the manifolds. This induces a more formidable knowledge distillation with smooth properties which preserves the past knowledge more efficiently as observed by our comprehensive empirical study. 1

Subfields: Representation & Self-Supervised Learning, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, methodology): “GeoDL [150] performs KD along a path that connects the low-dimensional projections of the old and new feature spaces for a smooth transition between them.”
RMM: Reinforced Memory Management for Class-Incremental Learning (2023, methodology): “Incremental Learning [22, 36, 48, 53, 58] continuously updates the model using data coming in a sequence of phases.”
Generative Negative Text Replay for Continual Vision-Language Pretraining (2022, methodology): “Existing literature focuses mostly on supervised continual learning [50,53,41], which mainly falls into the following four groups.”
Class-Incremental Learning with Cross-Space Clustering and Controlled Transfer (2022, methodology): “A number of these methods [42, 19, 13, 7, 2, 47, 8, 30] use variants of knowledge distillation [18], where the model and its predictions corresponding to the previous task are utilized to prevent the current task’s model from diverging too much from its previous state.”
Online Continual Learning with Contrastive Vision Transformer (2022, methodology): “The images used for training are randomly cropped and flipped for each method following [9,10,68].”
General Incremental Learning with Domain-aware Categorical Representations (2022, methodology): “We adopt the Replay, iCaRL [25], EE2L [4], UCIR [13], GeoDL [27], DER [37] and Meta-DR [33] as the comparison methods.”
Class-Incremental Learning by Knowledge Distillation with Adaptive Feature Consolidation (2022, methodology): “For comparisons, we report average incremental accuracy [37]—the average of the accuracies over seen classes across all incremental stages, where the results of other methods except GeoDL [44], DDE [17], and SSIL [2] are retrieved from [12] if not specified otherwise.”
Federated Class-Incremental Learning (2022, methodology): “In the standard class-incremental learning [37, 41, 43], there are a sequence of streaming tasks T = {T t } Tt =1 , where T denotes the task number, and the t -th task T t = { x ti , y ti } N t i =1 consists of N t pairs of samples x ti and their one-hot encoding labels y ti ∈ Y t .”
Self-Sustaining Representation Expansion for Non-Exemplar Class-Incremental Learning (2022, neutral): “However, most of the existing methods [21, 30] assume that a certain number (e.”
Representation Compensation Networks for Continual Semantic Segmentation (2022, neutral): “In the more challenging scenario without any replay, many methods explore regularization to maintain old knowledge, including knowledge distillation [10, 18, 21, 28, 51, 69, 74, 100], adversarial training [29,88], vanilla regularization [43,48,56,66,67,80,95,98] and so on.”
On Generalizing Beyond Domains in Cross-Domain Continual Learning (2022, methodology): “Concisely, LwF [23] applies knowledge distillation on the predictions, while LUCIR [16] employs preservation of features from old and current models, and GeoDL [30] extends the preservation in the feature space with smooth transitions of two subspaces from the old and the current models.”
Recent Advances of Continual Learning in Computer Vision: An Overview (2021, neutral): “[55] proposed to conduct knowledge distillation on the low-dimensional manifolds between the model outputs of previous and new tasks, which was shown to better mitigate catastrophic forgetting.”
Semantic-aware Knowledge Distillation for Few-Shot Class-Incremental Learning (2021, methodology): “In the literature [33], incremental learning techniques are categorized into three groups, task-incremental learning [4, 28, 21], domain-incremental learning [39, 29], and class-incremental learning [26, 1, 13, 36, 31].”
Continual Learning with Lifelong Vision Transformer (2022, methodology): “Continual learning baselines use ResNet18 [33] as backbone, and cross-entropy as classification loss, following [7, 10, 14, 15, 67, 71].”
Bring Evanescent Representations to Life in Lifelong Class Incremental Learning (2022, methodology): “Most of the successful CIL methods use exemplars of old classes C old to rehearse past knowledge [1, 3, 4, 7, 8, 12, 18, 20, 25, 27, 32, 36].”
SDDGR: Stable Diffusion-Based Deep Generative Replay for Class Incremental Object Detection (2024, methodology): “In CIL, we can cluster the main methods into knowledge distillation [16, 33, 38, 61], and re-play [5–7, 12, 17, 22, 47, 50, 57, 58, 66, 67] in general.”
Endpoints Weight Fusion for Class Incremental Semantic Segmentation (2023, background): “, knowledge distillation [39, 41] or gradient penalty [26, 29]).”
Class-Incremental Learning: A Survey (2023, background): “There are several ways to expand representation ability, and we divide them into three sub-groups, i.e. , neuron expansion , backbone expansion , and prompt expansion .”
Reinforced Attention for Few-Shot Learning and Beyond (2021, background): “The success of deep learning models rely heavily on a significant amount of labeled data, but the availability of large datasets is still limited due to the labor-intensive data preparation, which motivates the significant interest in fewshot learning [22, 40, 49, 11, 51, 47, 45].”