Preserving Earlier Knowledge in Continual Learning with the Help of All Previous Feature Extractors

By Zhuoyun Li, Changhong Zhong, Sijia Liu, Ruixuan Wang, Weishi Zheng. 2021

32 citations recorded by the source corpus.

Continual learning of new knowledge over time is one desirable capability for intelligent systems to recognize more and more classes of objects. Without or with very limited amount of old data stored, an intelligent system often catastrophically forgets previously learned old knowledge when learning new knowledge. Recently, various approaches have been proposed to alleviate the catastrophic forgetting issue. However, old knowledge learned earlier is commonly less preserved than that learned more recently. In order to reduce the forgetting of particularly earlier learned old knowledge and improve the overall continual learning performance, we propose a simple yet effective fusion mechanism by including all the previously learned feature extractors into the intelligent model. In addition, a new feature extractor is included to the model when learning a new set of classes each time, and a feature extractor pruning is also applied to prevent the whole model size from growing rapidly. Experiments on multiple classification tasks show that the proposed approach can effectively reduce the forgetting of old knowledge, achieving state-of-the-art continual learning performance.

Subfields: Artificial Intelligence, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Meta & Continual Learning

Citations in the corpus

S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learning (2022, methodology): “To overcome the second issue of the expansion-based methods, [68, 41] suggest learning a single classiﬁer on a concatenation of all resulting feature maps from different subsets of parameters, which connects all the tasks/domains.”
FOSTER: Feature Boosting and Compression for Class-Incremental Learning (2022, methodology): “Recently, methods [9,28,40] successfully apply the dynamic architectures into class incremental learning where the task id is unavailable, showing their advantages over the single backbone methods.”
DyTox: Transformers for Continual Learning with DYnamic TOken eXpansion (2021, methodology): “Baselines E2E [2] and Simple-DER [18] results come from their respective papers.”
A Model or 603 Exemplars: Towards Memory-Efficient Class-Incremental Learning (2022, background): “Rather than storing exemplars, recent works [54, 50, 30, 12] find that saving backbones from the history pushes the performance by one step towards the upper bound.”