Continual Learning in Deep Neural Network by Using a Kalman Optimiser

By Honglin Li, Shirin Enshaeifar, Frieder Ganz, Payam M. Barnaghi. 2019

5 citations recorded by the source corpus.

Learning and adapting to new distributions or learning new tasks sequentially without forgetting the previously learned knowledge is a challenging phenomenon in continual learning models. Most of the conventional deep learning models are not capable of learning new tasks sequentially in one model without forgetting the previously learned ones. We address this issue by using a Kalman Optimiser. The Kalman Optimiser divides the neural network into two parts: the long-term and short-term memory units. The long-term memory unit is used to remember the learned tasks and the short-term memory unit is to adapt to the new task. We have evaluated our method on MNIST, CIFAR10, CIFAR100 datasets and compare our results with state-of-the-art baseline models. The results show that our approach enables the model to continually learn and adapt to the new changes without forgetting the previously learned tasks.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Richard E. Turner
2. Tameem Adel
3. Han Zhao
4. Siddharth Swaroop
5. Thang D. Bui
6. Yarin Gal
7. Yingzhen Li
8. Jun Zhu

Citations in the corpus

• Continual Learning with Adaptive Weights (CLAW) (2019, methodology): “(b) The architecture-based approach aims to deal with stability and adaptation issues by a fixed division of the architecture into global and local parts (Rusu et al., 2016b; Fernando et al., 2017; Shin et al., 2017; Kaplanis et al., 2018; Xu & Zhu, 2018; Yoon et al., 2018; Li et al., 2019b).”