A comprehensive, application-oriented study of catastrophic forgetting in DNNs

By Benedikt Pfülb, Alexander Rainer Tassilo Gepperth. 2019

95 citations recorded by the source corpus.

We present a large-scale empirical study of catastrophic forgetting (CF) in modern Deep Neural Network (DNN) models that perform sequential (or: incremental) learning. A new experimental protocol is proposed that enforces typical constraints encountered in application scenarios. As the investigation is empirical, we evaluate CF behavior on the hitherto largest number of visual classification datasets, from each of which we construct a representative number of Sequential Learning Tasks (SLTs) in close alignment to previous works on CF. Our results clearly indicate that there is no model that avoids CF for all investigated datasets and SLTs under application conditions. We conclude with a discussion of potential solutions and workarounds to CF, notably for the EWC and IMM models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Meta & Continual Learning

Citations in the corpus

Continual Learning with Fully Probabilistic Models (2021, background): “See [6] for a longer discussion on this point.”
Continual Lifelong Learning in Natural Language Processing: A Survey (2020, methodology): “…methodologies for CL method evaluation have been devised throughout the years (e.g. Kemker et al., 2017; Serra et al., 2018; Sodhani et al., 2020; Pfülb and Gepperth, 2019; Chaudhry et al., 2019a); however, many of them suffer from deficiencies such as small datasets or a limited number of…”
Continual Learning With Extended Kronecker-Factored Approximate Curvature (2020, background): “However, as [29] pointed out, many approaches do not explain why hyperparameters were selected to have such specific values and how to select them in a real-world case, or they presciently use the best hyperparameters selected after all tasks have been processed, which can implicitly violate causali”
iTAML: An Incremental Task-Agnostic Meta-learning Approach (2020, background): “Despite several attempts, catastrophic forgetting and interference are still challenging problems for deep learning [20].”
Continual Learning with Adaptive Weights (CLAW) (2019, background): “Indeed, catastrophic forgetting is one of the main pathologies in continual learning (McCloskey and Cohen, 1989; Ratcliff, 1990; Robins, 1993, 1995; French, 1999; Pape et al., 2011; Goodfellow et al., 2014a; Achille et al., 2018; Kemker et al., 2018; Kemker and Kanan, 2018; Diaz-Rodriguez et al., 20”
Continual learning: A comparative study on how to defy forgetting in classification tasks (2019, background): “Pfülb and Gepperth [26] perform an empirical study on catastrophic forgetting and develop a protocol for setting hyperparameters and method evaluation.”
Class-Incremental Learning: Survey and Performance Evaluation on Image Classification (2020, methodology): “Pfulb and Gepperth [62] propose a training and evaluation paradigm for task-IL methods, limited to two tasks.”
Continual Learning for Robotics (2019, background): “the performance obtained on task Tt to fine-tune a model that will learn on T0 violates temporal causality (Pfulb and Gepperth, 2019).”
A Wholistic View of Continual Learning with Deep Neural Networks: Forgotten Lessons and the Bridge to Active and Open World Learning (2020, methodology): “Empirical surveys in the context of robotics (Lesort et al., 2019), generative models (Lesort et al., 2020) follow similar trends and conduct a “comprehensive application-oriented study of catastrophic forgetting” (Pfülb and Gepperth, 2019).”
Marginal Replay vs Conditional Replay for Continual Learning (2018, background): “In fact, recent articles [29] show that MNIST-related CLTs are still a major obstacle for most current approaches to continual learning under realistic conditions.”
A Continual Learning Survey: Defying Forgetting in Classification Tasks (2019, background): “Pf¨ulb and Gepperth [25] perform an empirical study on catastrophic forgetting and develop a protocol for setting hyperparameters and method evaluation.”