Regularize, Expand and Compress: NonExpansive Continual Learning

By Jie Zhang, Junting Zhang, Dawei Li, Jingwen Zhu, Heming Zhang, Yalin Wang. 2020

27 citations recorded by the source corpus.

Continual learning (CL), the problem of lifelong learning where tasks arrive in sequence, has attracted increasing attention in the computer vision community lately. The goal of CL is to learn new tasks while maintaining the performance on the previously learned tasks. There are two major obstacles for CL of deep neural networks: catastrophic forgetting and limited model capacity. Inspired by the recent breakthroughs in automatically learning good neural network architectures, we develop a nonexpansive AutoML framework for CL termed Regularize, Expand and Compress (REC) to solve the above issues. REC is a unified framework with three highlights: 1) a novel regularized weight consolidation (RWC) algorithm to avoid forgetting, where accessing the data seen in the previously learned tasks is not required; 2) an automatic neural architecture search (AutoML) engine to expand the network to increase model capability; 3) smart compression of the expanded model after a new task is learned to improve the model efficiency. The experimental results on four different image recognition datasets demonstrate the superior performance of the proposed REC over other CL algorithms.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Class-incremental Learning via Deep Model Consolidation (2019, background): “ incrementally training generative models [21,50], which is a harder problem to solve; while others ﬁne-tune the old model on the new data with certain regularization techniques to prevent forgetting [1,6,23,29,51,57,59]. We argue that the ineffectiveness of these regularization-based methods is mai”
Efficient Architecture Search for Continual Learning (2020, unknown): “By assuming some extent of correlation between consecutive tasks, it uses regularization terms to prevent large deviation of the network weights when reoptimized [23].”