Overcoming Catastrophic Forgetting for Continual Learning via Model Adaptation

By Wenpeng Hu, Bing Liu, Chongyang Tao, Zhengwei Tao, Jinwen Ma, Dongyan Zhao, Rui Yan. 2018

150 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “In a broader sense, sampling parameters from taskconditioned parameter distributions (MERLIN [221], PR [160], PGMA [169], HNET [420]), as well as stabilizing important parameters with weight regularization, can be seen as a form of deriving task-specific/adaptive parameters.”
Learnability and Algorithm for Continual Learning (2023, methodology): “Pseudoreplay methods generate pseudo replay data for old tasks to serve as the replay data (Kamra et al., 2017; Shin et al., 2017; Wu et al., 2018; Seff et al., 2017; Kemker & Kanan, 2018; Hu et al., 2019; Rostami et al., 2019; Ostapenko et al., 2019).”
RMM: Reinforced Memory Management for Class-Incremental Learning (2023, background): “Recent papers are either task-incremental learning—each phase corresponds to a task (dataset) that contains new data of all seen classes [7, 10, 19, 29, 43, 47, 57], or class-incremental learning (CIL)—each phase contains data of a new set of classes, i.e., classes are unseen [4, 6, 18, 25, 31, 37,…”
A Theoretical Study on Solving Continual Learning (2022, methodology): “Using regularization [12, 13] to minimize changes to model parameters learned from previous tasks is a popular approach [14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26].”
S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learning (2022, methodology): “For instance, memory-based methods alleviate forgetting by either replaying a small set of examples from past tasks saved in a memory [57, 55, 46, 7, 23, 28, 9] or replaying synthesized data from previous tasks using generative models [60, 63].”
Adaptive Orthogonal Projection for Batch and Online Continual Learning (2022, methodology): “ThecodeofPGMA:https://github.com/morning-dews/PGMA_ tensorﬂow.”
Overcoming Catastrophic Forgetting in Incremental Few-Shot Learning by Finding Flat Minima (2021, methodology): “Previous research mainly includes multi-class incremental learning [4, 38, 22, 33, 54, 51] and multi-task incremental learning [21, 31, 42].”
Flattening Sharpness for Dynamic Gradient Projection Memory Benefits Continual Learning (2021, methodology): “This involves using knowledge distillation [23, 14] or penalizing changes in weights deemed important for previous tasks [17, 46, 28, 2, 3] to reduce forgetting.”
Recent Advances of Continual Learning in Computer Vision: An Overview (2021, neutral): “[169] proposed a Parameter Generation and Model Adaptation (PGMA) method, in which the model parameters are dynamically changed at inference time instead of training time.”
Continual Learning for Named Entity Recognition (2021, methodology): “Hu et al. (2019) introduce the Parameter Generation and Model Adaptation framework for image and text classiﬁers.”
Continual Learning by Using Information of Each Class Holistically (2021, methodology): “The third approach builds data generators to generate pseudo-examples for old tasks to be used used in learning the new task (Shin et al. 2017; Kamra, Gupta, and Liu 2017; Seff et al. 2017; Hu et al. 2019).”
Training Networks in Null Space of Feature Covariance for Continual Learning (2021, neutral): “An alternative approach is to learn generative model to generate synthetic data to substitute the original data [16, 19, 37, 46].”
Adaptive Aggregation Networks for Class-Incremental Learning (2020, background): “, all-class data come but are from a different dataset for each new phase [5,6,8,17,23,35,40,54], or class-based i.”
Routing Networks with Co-training for Continual Learning (2020, methodology): “The PGMA method introduced by Hu et al. (2019) splits the network parameters in two parts; one part that is shared among all tasks and another part that gets adapted to the test instance in order to classify it.”
Mnemonics Training: Multi-Class Incremental Learning Without Forgetting (2020, background): “Recent works are either in the multi-task setting (classes from different datasets) [16, 27, 10, 4, 25], or in the multi-class setting (classes from the identical dataset) [24, 9, 34, 2].”
Dreaming to Distill: Data-Free Knowledge Transfer via DeepInversion (2019, background): “…be obtained from the original dataset, e.g ., a subset of data (iCaRL [56]), parameter importance estimations (in the form of Fisher matrix in EWC [27], contribution to loss change in SI [74], posterior of network weights in VCL [50]), or training data representation (encoder [62], GAN [21, 59]).”
Bayesian Structure Adaptation for Continual Learning (2019, background): “…optimization problem at each task, the constraint being that the loss should decrease monotonically on a heuristically selected replay buffer; (Hu et al., 2018), which uses a partially shared parameter space for inter-task transfer and generates the replay samples through a data-generative…”
Orthogonal Gradient Descent for Continual Learning (2019, methodology): “…proposed a cycle of active learning (progression) followed by consolidation (compression) that requires no architecture growth and no access to or storing of previous data or tasks which is similar in spirit to distillation based methods of continual learning Li and Hoiem (2017); Hu et al. (2018).”