Keep and Learn: Continual Learning by Constraining the Latent Space for Knowledge Preservation in Neural Networks

By Hyo-Eun Kim, Seungwook Kim, Jaehwan Lee. 2018

31 citations recorded by the source corpus.

Data is one of the most important factors in machine learning. However, even if we have high-quality data, there is a situation in which access to the data is restricted. For example, access to the medical data from outside is strictly limited due to the privacy issues. In this case, we have to learn a model sequentially only with the data accessible in the corresponding stage. In this work, we propose a new method for preserving learned knowledge by modeling the high-level feature space and the output space to be mutually informative, and constraining feature vectors to lie in the modeled space during training. The proposed method is easy to implement as it can be applied by simply adding a reconstruction loss to an objective function. We evaluate the proposed method on CIFAR-10/100 and a chest X-ray dataset, and show benefits in terms of knowledge preservation compared to previous approaches.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

In Meta & Continual Learning

Citations in the corpus

Continual Learning via Sequential Function-Space Variational Inference (2023, background): ““Keep and learn” (Kim et al., 2018) also uses internal representations as a basis for regularization.”
Continual Learning with Adaptive Weights (CLAW) (2019, methodology): “…radical changes, while allowing the rest of the parameters to change without restriction (Li & Hoiem, 2016; Lee et al., 2017; Zenke et al., 2017; Chaudhry et al., 2018; Kim et al., 2018; Nguyen et al., 2018; Srivastava et al., 2013; Schwarz et al., 2018; Vuorio et al., 2018; Aljundi et al., 2019c).”
Overcoming Catastrophic Forgetting by Neuron-level Plasticity Control (2019, methodology): “(Li and Hoiem 2018; Shin et al. 2017; Kamra, Gupta, and Liu 2017; Zacarias and Alexandre 2018; Kim, Kim, and Lee 2018) Another approach is to isolate the parts of the neural network containing previous knowledge, and learn the new task using other parts of the network.”
Overcoming Catastrophic Forgetting With Unlabeled Data in the Wild (2019, methodology): “Note that the model-based and data-based approaches are orthogonal in most cases, thus they can be combined for better performance [15].”
A comprehensive, application-oriented study of catastrophic forgetting in DNNs (2019, background): “Most approaches ignore model selection (Lee et al., 2017; Srivastava et al., 2013; Aljundi et al., 2018; Kim et al., 2018), and thus implicitly violate causality.”
Marginal Replay vs Conditional Replay for Continual Learning (2018, background): “3, 6], dual-memory systems [15, 32, 8], distillation-based approaches [39, 24, 17] and generative replay methods [38, 15, 22, 14, 42].”
Incremental Learning with Unlabeled Data in the Wild (2019, methodology): “e model and replay the generated data when training a new model. Note that the model-based and the data-based approaches are orthogonal in most cases, thus they can be combined for better performance [12]. Knowledge distillation in prior works. Our proposed method is a data-based approach, but it is”