Bi-Objective Continual Learning: Learning 'New' While Consolidating 'Known'

By Xiaoyu Tao, Xiaopeng Hong, Xinyuan Chang, Yihong Gong. 2020

31 citations recorded by the source corpus.

In this paper, we propose a novel single-task continual learning framework named Bi-Objective Continual Learning (BOCL). BOCL aims at both consolidating historical knowledge and learning from new data. On one hand, we propose to preserve the old knowledge using a small set of pillars, and develop the pillar consolidation (PLC) loss to preserve the old knowledge and to alleviate the catastrophic forgetting problem. On the other hand, we develop the contrastive pillar (CPL) loss term to improve the classification performance, and examine several data sampling strategies for efficient onsite learning from ‘new’ with a reasonable amount of computational resources. Comprehensive experiments on CIFAR10/100, CORe50 and a subset of ImageNet validate the BOCL framework. We also reveal the performance accuracy of different sampling strategies when used to finetune a given CNN model. The code will be released.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Meta & Continual Learning

Citations in the corpus

Adaptive Orthogonal Projection for Batch and Online Continual Learning (2022, methodology): “Knowledge distillation , which is a kind of regularization, has also been commonly used (Li and Hoiem 2017; Wu et al. 2019; Castro et al. 2018; Belouadah and Popescu 2019; Liu et al. 2020; Lee et al. 2019; Tao et al. 2020).”
Continual Learning by Using Information of Each Class Holistically (2021, methodology): “Knowledge distillation was used by LwF (Li and Hoiem 2017) and others (Wu et al. 2019; Castro et al. 2018; Be-louadah and Popescu 2019; Liu et al. 2020; Lee et al. 2019; Tao et al. 2020).”
Few-Shot Class-Incremental Learning via Relation Knowledge Distillation (2021, methodology): “In the C-way K-shot FSCIL setting (Tao et al. 2020b; Zhao et al. 2020), the number of classes in C ( t ) is C and the number of training samples per class is K for each X ( t ) ; t > 1 .”
Few-Shot Class-Incremental Learning (2020, background): “Recently, there have been many research efforts attempting to solve the catastrophic forgetting problem [15, 49, 20, 24, 18, 32, 2, 13, 41, 37, 1].”
Class-Incremental Learning with Topological Schemas of Memory Spaces (2021, methodology): “Compared with the new-instances incremental learning [34], [35] setting, a more typical SIT setting is the class-incremental learning focused by our paper.”
Balancing Between Forgetting and Acquisition in Incremental Subpopulation Learning (2022, methodology): “Among them, EWC and LwF are widely used to benchmark various IL settings including IDL [12,32,31,33] and achieve comparable results to the state-of-the-art (SOTA).”
Class Incremental Learning for Video Action Classification (2021, background): “There are three main directions to overcome catastrophic forgetting: (1) To regularize the important parameters of the old model when training new data [9, 10, 11, 12], (2) To retain some old samples for knowledge distillation technology [13, 14, 15, 16, 17], (3) To generate old samples based on gen”
BNS: Building Network Structures Dynamically for Continual Learning (2021, background): “Many techniques also use knowledge distillation [33, 59, 6, 4, 35, 29, 53], which is another family of methods that relies on regularization, i.”
Topology-Preserving Class-Incremental Learning (2020, methodology): “The multi-task incremental learning [30, 19] aims at learning a sequence of independent tasks, each of which is assigned a speciﬁc classiﬁer, while the single-task incremental learning [27, 38] employs a uniﬁed classiﬁer to treat the entire incremental learning process as one task.”