Less-forgetful Learning for Domain Expansion in Deep Neural Networks

By Heechul Jung, Jeongwoo Ju, Minju Jung, Junmo Kim. 2017

78 citations recorded by the source corpus.

Expanding the domain that deep neural network has already learned without accessing old domain data is a challenging task because deep neural networks forget previously learned information when learning new data from a new domain. In this paper, we propose a less-forgetful learning method for the domain expansion scenario. While existing domain adaptation techniques solely focused on adapting to new domains, the proposed technique focuses on working well with both old and new domains without needing to know whether the input is from the old or new domain. First, we present two naive approaches which will be problematic, then we provide a new method using two proposed properties for less-forgetful learning. Finally, we prove the effectiveness of our method through experiments on image classification tasks. All datasets used in the paper, will be released on our website for someone's follow-up study.

Subfields: Representation & Self-Supervised Learning, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

In Meta & Continual Learning

Citations in the corpus

Continual Learning via Sequential Function-Space Variational Inference (2023, methodology): ““Less-forgetful learning” (Jung et al., 2018) employs the same method but uses squared Euclidean distance rather than the modified cross-entropy loss and applies it to the penultimate-layer representations rather than the network’s predictions.”
Online Continual Learning in Image Classification: An Empirical Survey (2021, methodology): “Compared tricks To mitigate the strong bias towards new classes due to class imbalance, multiple methods have been proposed lately [98, 99, 68, 70, 100].”
Generative Feature Replay For Class-Incremental Learning (2020, methodology): “Instead of regularizing weights, [15, 23] align the predictions using the data from the current task.”
Continual Learning by Asymmetric Loss Approximation With Single-Side Overestimation (2019, background): “The functional approaches [23, 14, 32, 28] often incorporate knowledge distillation to establish continual learning.”
Overcoming Catastrophic Forgetting With Unlabeled Data in the Wild (2019, background): “Model-based approaches [1, 4, 14, 16, 21, 25, 27, 30, 34, 35, 36, 37, 43, 45] keep the knowledge of previous tasks by penalizing the change of parameters crucial for previous tasks, i.e., the updated parameters are constrained to be around the original values, and the update is scaled down by the…”
Class-incremental Learning with Pre-allocated Fixed Classifiers (2020, background): “This was exploited in [40], [41] to reduce catastrophic forgetting in domain adaptation.”
Lifelong Learning via Progressive Distillation and Retrospection (2018, methodology): “[12] propose to approximate the features of the original network for old tasks rather than the output of the last layer.”
Learning an Evolved Mixture Model for Task-Free Continual Learning (2022, background): “branches: regularization-based [8], generative replay mechanism [4, 9] and dynamic expansion approaches [10, 11].”
Class-Incremental Learning: A Survey (2023, background): “Several works follow it to utilize feature distillation in CIL [90], [91], [97], [98], while others address distilling other products.”
Efficient Continual Learning in Neural Networks with Embedding Regularization (2019, methodology): “The method we propose is connected to the less-forgetful learning (LFL) [12] method in domain adaptation, and in a lesser form to neural graph machines [4] for graph-based learning.”
An End-to-End Architecture for Class-Incremental Object Detection with Knowledge Distillation (2019, methodology): “[13] expanded the domain of old classes to include the domain of new classes, but both old and new domains should be similar or share significant overlap.”
Learning a Unified Classifier Incrementally via Rebalancing (2019, methodology): “However, our method differs from [19] in three aspects.”
Regularize, Expand and Compress: NonExpansive Continual Learning (2020, background): “Less-forgetful learning [14] is proposed to regularize the L 2 distance between the ﬁnal hidden activations and the old tasks’ parameters for preserving the old task feature mappings.”
CILEA-NET: Curriculum-Based Incremental Learning Framework for Remote Sensing Image Classification (2021, unknown): “While[7]and[22]alignsthepredictionsbased onthecurrenttasksathand.”
Incremental Learning with Unlabeled Data in the Wild (2019, background): “e summarized as continual learning, and recent works on continual learning have studied two types of approaches to overcome catastrophic forgetting: model-based and data-based. Model-based approaches [1,4,14,16,21,25,27,30,34,35,36,37,43,45] keep the knowledge of previous tasks by penalizing the cha”
Striking a Balance between Stability and Plasticity for Class-Incremental Learning (2021, methodology): “To transfer knowledge from an old model to a new model, we constrain the distance between normalized embeddings [19, 39] from the new model ( φ ( x ) n ) and the old model ( φ ( x ) o ) as the embedding supervision loss L em , i.e ., Therefore, the overall training objective L can be deﬁned as L =…”