Understanding Catastrophic Forgetting and Remembering in Continual Learning with Optimal Relevance Mapping

By Prakhar Kaushik, Alex Gain, Adam Kortylewski, Alan L. Yuille. 2021

80 citations recorded by the source corpus.

Catastrophic forgetting in neural networks is a significant problem for continual learning. A majority of the current methods replay previous data during training, which violates the constraints of an ideal continual learning system. Additionally, current approaches that deal with forgetting ignore the problem of catastrophic remembering, i.e. the worsening ability to discriminate between data from different tasks. In our work, we introduce Relevance Mapping Networks (RMNs) which are inspired by the Optimal Overlap Hypothesis. The mappings reflects the relevance of the weights for the task at hand by assigning large weights to essential parameters. We show that RMNs learn an optimized representational overlap that overcomes the twin problem of catastrophic forgetting and remembering. Our approach achieves state-of-the-art performance across all common continual learning datasets, even significantly outperforming data replay methods while not violating the constraints for an ideal continual learning system. Moreover, RMNs retain the ability to detect data from new tasks in an unsupervised manner, thus proving their resilience against catastrophic remembering.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Continual Learning with Recursive Gradient Optimization (2022, background): “Expansionbased approaches select the network parameters dynamically (Yoon et al., 2018; Rosenbaum et al., 2018; Serra et al., 2018; Kaushik et al., 2021), add additional components as new tasks arrive (Rusu et al., 2016; Fernando et al., 2017; Alet et al., 2018; Chang et al., 2019; Li et al.,…”
Model Zoo: A Growing Brain That Learns Continually (2021, background): “This indicates that replaying data from past tasks is beneﬁcial (Robins, 1995), even if replay may not conform to certain stylistic formulations of continual learning in the literature (Farquhar and Gal, 2019a; Kaushik et al., 2021).”
Measuring Forgetting of Memorized Training Examples (2022, background): “Our definition of forgetting is inspired by the widely observed phenomenon of catastrophic forgetting [22, 54, 45, 43, 16, 42], where a model tends to forget previously learned knowledge when training on new data.”
VariGrow: Variational Architecture Growing for Task-Agnostic Continual Learning based on Bayesian Novelty (2022, methodology): “Some methods proposed to tackle the task-agnostic setting, but only during testing (Kaushik et al., 2021; Yan et al., 2021b; Abati et al., 2020; Rajasegaran et al., 2020).”
Provable and Efficient Continual Representation Learning (2022, methodology): “In our work, consistent with our theory, we focus on zero-forgetting CL [54, 33, 53, 84, 35, 77], which is a sub-branch of architecture-based methods and completely eliminates forgetting.”