Bayesian Optimized Continual Learning with Attention Mechanism

By Ju Xu, Jin Ma, Zhanxing Zhu. 2019

7 citations recorded by the source corpus.

Though neural networks have achieved much progress in various applications, it is still highly challenging for them to learn from a continuous stream of tasks without forgetting. Continual learning, a new learning paradigm, aims to solve this issue. In this work, we propose a new model for continual learning, called Bayesian Optimized Continual Learning with Attention Mechanism (BOCL) that dynamically expands the network capacity upon the arrival of new tasks by Bayesian optimization and selectively utilizes previous knowledge (e.g. feature maps of previous tasks) via attention mechanism. Our experiments on variants of MNIST and CIFAR-100 demonstrate that our methods outperform the state-of-the-art in preventing catastrophic forgetting and fitting new tasks better.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Richard E. Turner
2. Tameem Adel
3. Han Zhao
4. Siddharth Swaroop
5. Thang D. Bui
6. Yarin Gal
7. Yingzhen Li
8. Jun Zhu

Citations in the corpus

• Continual Learning with Adaptive Weights (CLAW) (2019, background): “…and adaptation issues via dividing the architecture into reusable parts that are less prone to changes, and other parts especially devoted to individual tasks (Rusu et al., 2016b; Fernando et al., 2017; Yoon et al., 2018; Du et al., 2019; He et al., 2019; Li et al., 2019a; Xu et al., 2019).”