Pseudo-Recursal: Solving the Catastrophic Forgetting Problem in Deep Neural Networks

By Craig Atkinson, Brendan McCane, Lech Szymanski, Anthony V. Robins. 2018

78 citations recorded by the source corpus.

In general, neural networks are not currently capable of learning tasks in a sequential fashion. When a novel, unrelated task is learnt by a neural network, it substantially forgets how to solve previously learnt tasks. One of the original solutions to this problem is pseudo-rehearsal, which involves learning the new task while rehearsing generated items representative of the previous task/s. This is very effective for simple tasks. However, pseudo-rehearsal has not yet been successfully applied to very complex tasks because in these tasks it is difficult to generate representative items. We accomplish pseudo-rehearsal by using a Generative Adversarial Network to generate items so that our deep network can learn to sequentially classify the CIFAR-10, SVHN and MNIST datasets. After training on all tasks, our network loses only 1.67% absolute accuracy on CIFAR-10 and gains 0.24% absolute accuracy on SVHN. Our model's performance is a substantial improvement compared to the current state of the art solution.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

In Meta & Continual Learning

Citations in the corpus

NISPA: Neuro-Inspired Stability-Plasticity Adaptation for Continual Learning in Sparse Networks (2022, methodology): “, 2019; Isele & Cosgun, 2018) or generate (Shin et al., 2017; Atkinson et al., 2018; Ramapuram et al., 2020) examples of previous tasks.”
Continual Sequence Generation with Adaptive Compositional Modules (2022, background): “…privacy concerns, this work also follows a line of work that doesn’t store real examples for experience replay, such as generating examples by GAN (Atkinson et al., 2018), synthesizing examples (Xu et al., 2022) by model-inversion (Smith et al., 2021b), and using unlabeled data in the learning…”
Replay in Deep Learning: Current Approaches and Missing Biological Elements (2021, neutral): “…adversarial network (GAN) (Goodfellow et al., 2014) to generate samples from previously learned data (Draelos et al., 2017; Kemker & Kanan, 2018; Robins, 1995, 1996; Ostapenko et al., 2019; Shin, Lee, Kim, & Kim, 2017; He et al., 2018; French, 1997; Atkinson, McCane, Szymanski, & Robins, 2018).”
Continual learning: A comparative study on how to defy forgetting in classification tasks (2019, methodology): “Rehearsal iCaRL [18] ER [44] SER [45] TEM [46] DGR [14] PR [47] CCLUGM [48] LGM [49] Constrained GEM [50] A-GEM [8] GSS [43] Regularization-basedmethods Prior-focused EWC [28] IMM [29] SI [51] R-EWC [52] MAS [15] [16] Data-focused LwF [53] LFL [54] EBLL [11] DMC [55] Parameter isolation methods”
Generative replay with feedback connections as a general strategy for continual learning (2018, background): “This approach had some success with very simple, artificial inputs, but does not work with more complicated inputs [11].”
Variational Prototype Replays for Continual Learning (2019, background): “These generative approaches have succeeded over very simple and artiﬁcial inputs but they cannot tackle more complicated inputs (Atkinson et al., 2018).”
A Continual Learning Survey: Defying Forgetting in Classification Tasks (2019, methodology): “With deep networks and large input vectors (e.g. full resolution images), random input cannot cover the input space [52].”
Mix-review: Alleviate Forgetting in the Pretrain-Finetune Framework for Neural Language Generation Models (2019, unknown): “Note that the forgetting phenomenon here is not necessarily “catastrophic” as in the sequential learning case (Atkinson et al., 2018; Robins, 1995), because the goal is to achieve the best performance on the target dialogue data-set, and the model does not need to maintain fidelity to the pre-traini”
Pseudo-Rehearsal: Achieving Deep Reinforcement Learning without Catastrophic Forgetting (2018, background): “Although pseudo-rehearsal works for neural networks with relatively small input spaces, it does not scale well to datasets with large input spaces such as image datasets [15].”
Efficient Continual Learning in Neural Networks with Embedding Regularization (2019, background): “Another class of algorithms, known as rehearsal methods [23, 28, 2], keep a memory of data analyzed on previous tasks, and continually retrain the network on this memory in order to maintain its performance.”
Automatic Recall Machines: Internal Replay, Continual Learning and the Brain (2020, background): “…either in the form of buffers [Chaudhry et al., 2019, Aljundi et al., 2019a,b, Riemer et al., 2018, Rebuffi et al., 2017, Rolnick et al., 2019] or dedicated generators [Shin et al., 2017, Kamra et al., 2017, Lavda et al., 2018, Lesort et al., 2019b, Atkinson et al., 2018, Kemker and Kanan, 2017].”
Prototype Reminding for Continual Learning (2019, background): “These generative approaches have succeeded over very simple and artificial inputs but they cannot tackle more complicated inputs [2].”
Analyzing the Forgetting Problem in the Pretrain-Finetuning of Dialogue Response Models (2019, unknown): “onse problem a more relevant issue to solve (He & Glass, 2019b). 7 RELATED WORKS Forgetting As discussed in Section 3.2, in contrast to the “catastrophic forgetting” problem in sequentiallearning(Atkinson et al., 2018; Robins,1995; Riemer et al.,2017), theperformancedrop on pre-training data is ”
Analyzing the Forgetting Problem in Pretrain-Finetuning of Open-domain Dialogue Response Models (2021, background): “Forgetting As discussed in Section 3.2, in contrast to the “catastrophic forgetting” problem in sequential learning (Atkinson et al., 2018; Robins, 1995), the performance drop on pretraining data is not necessarily bad for the NLP pretrain-ﬁnetune framework, and its implications have not been…”
Network Adaptation Strategies for Learning New Classes without Forgetting the Original Ones (2019, methodology): “Other approaches train generative models using the samples from the first phase, and use them to generate samples in the retraining phase [7, 8, 9].”
Towards Knowledgeable Supervised Lifelong Learning Systems (2020, methodology): “Pseudo-recursal (Atkinson et al., 2018a) used Generative Adversarial Networks (GAN) (Goodfellow, Pouget-Abadie, Mirza, Xu, Warde-Farley, Ozair, Courville, & Bengio, 2014) to generate examples from previous tasks used to retrain old tasks while learning a new task.”