Single-Net Continual Learning with Progressive Segmented Training

By Xiaocong Du, Gouranga Charan, Frank Liu, Yu Cao. 2019

13 citations recorded by the source corpus.

There is an increasing need of continual learning in dynamic systems, such as the self-driving vehicle, the surveillance drone, and the robotic system. Such a system requires learning from the data stream, training the model to preserve previous information and adapt to a new task, and generating a single-headed vector for future inference. Different from previous approaches with dynamic structures, this work focuses on a single network and model segmentation to prevent catastrophic forgetting. Leveraging the redundant capacity of a single network, model parameters for each task are separated into two groups: one important group which is frozen to preserve current knowledge, and secondary group to be saved (not pruned) for a future learning. A fixed-size memory containing a small amount of previously seen data is further adopted to assist the training. Without additional regularization, the simple yet effective approach of Progressive Segmented Training (PST) successfully incorporates multiple tasks and achieves the state-of-the-art accuracy in the single-head evaluation on CIFAR-10 and CIFAR-100 datasets. Moreover, the segmented training significantly improves computation efficiency in continual learning at the edge.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Richard E. Turner
2. Tameem Adel
3. Han Zhao
4. Siddharth Swaroop
5. Thang D. Bui
6. Yarin Gal
7. Yingzhen Li
8. Jun Zhu

In Computer Vision and Pattern Recognition

1. Tameem Adel
2. Richard E. Turner
3. Han Zhao
4. Yarin Gal
5. Frederik Benzing
6. Qiang Liu
7. Max Welling
8. Fei Ye

Citations in the corpus

• Continual Learning with Adaptive Weights (CLAW) (2019, background): “…and adaptation issues via dividing the architecture into reusable parts that are less prone to changes, and other parts especially devoted to individual tasks (Rusu et al., 2016b; Fernando et al., 2017; Yoon et al., 2018; Du et al., 2019; He et al., 2019; Li et al., 2019a; Xu et al., 2019).”