Continual Learning by Using Information of Each Class Holistically

By Wenpeng Hu, Qi Qin, Mengyu Wang, Jinwen Ma, Bing Liu. 2021

66 citations recorded by the source corpus.

Continual learning (CL) incrementally learns a sequence of tasks while solving the catastrophic forgetting (CF) problem. Existing methods mainly try to deal with CF directly. In this paper, we propose to avoid CF by considering the features of each class holistically rather than only the discriminative information for classifying the classes seen so far. This latter approach is prone to CF because the discriminative information for old classes may not be sufficiently discriminative for the new class to be learned. Consequently, in learning each new task, the network parameters for previous tasks have to be revised, which causes CF. With the holistic consideration, after adding new tasks, the system can still do well for previous tasks. The proposed technique is called Per-class Continual Learning (PCL). PCL has two key novelties. (1) It proposes a one-class learning based technique for CL, which considers features of each class holistically and represents a new approach to solving the CL problem. (2) It proposes a method to extract discriminative information after training to further improve the accuracy. Empirical evaluation shows that PCL markedly outperforms the state-of-the-art baselines for one or more classes per task. More tasks also result in more gains.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Bing Liu
2. Zixuan Ke
3. Hu Xu
4. Lei Shu
5. B. Liu
6. Bin Liu
7. Nianzu Ma
8. Gyuhak Kim

In Computer Vision and Pattern Recognition

1. Zixuan Ke
2. Bin Liu
3. Bing Liu
4. Dongyan Zhao
5. Hu Xu
6. Gyuhak Kim
7. Hang Su
8. B. Ermiş

In Meta & Continual Learning

1. Bing Liu
2. Zixuan Ke
3. Bin Liu
4. Gyuhak Kim
5. Hu Xu
6. Lei Shu
7. Dongyan Zhao
8. B. Liu

Citations in the corpus

• A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, methodology): “Feature Space LUCIR [164], PODNet [100], TPCIL [403], PCL [170], AANets [270], DER [468], DDE [171], GeoDL [387], PASS [510], Co2L [57], AFC [199], SP-CIL [456], ELI [192], CwD [380], CSCCT [22], FOSTER [424], FASP [300], CLS-ER [18]”
• Adaptive Orthogonal Projection for Batch and Online Continual Learning (2022, background): “Many other papers use related approaches (Zenke, Poole, and Ganguli 2017; Fernando et al. 2017; Aljundi et al. 2017; Ritter, Botev, and Barber 2018; Xu and Zhu 2018; Kemker and Kanan 2018; Parisi et al. 2018; Ahn et al. 2019; Hu et al. 2021; Dhar et al. 2019; Adel, Zhao, and Turner 2020).”
• Memory Efficient Continual Learning with Transformers (2022, background): “[23] Wenpeng Hu, Qi Qin, Mengyu Wang, Jinwen Ma, and Bing Liu.”
• An Empirical Investigation of the Role of Pre-training in Lifelong Learning (2021, background): “Whereas starting from a random initialization may provide a more challenging learning environment, it is simply not practical in the context of deploying state-of-the-art systems for real-world tasks.”
• Achieving Forgetting Prevention and Knowledge Transfer in Continual Learning (2021, methodology): “But such pre-trained models or feature extractors can significantly improve the CL performance [18, 24].”
• Recent Advances of Continual Learning in Computer Vision: An Overview (2021, neutral): “[34] proposed Per-class Continual Learning (PCL) which treats every task holistically on itself.”
• BNS: Building Network Structures Dynamically for Continual Learning (2021, methodology): “Recently, pre-trained feature extractors have also been employed to improve the CL accuracy [19, 22].”
• Memory Efficient Continual Learning for Neural Text Classification (2022, background): “Recently, some studies use pre-trained models for class incremental learning (Ke et al., 2021; Hu et al., 2021); (4) Parameter isolation-based methods (Ke et al., 2020; Worts-man et al., 2020) allocate model parameters dedicated to different tasks and mask them out when learning a new task; (5)…”
• Online Continual Learning through Mutual Information Maximization (2022, methodology): “Wang et al. (2021) proposed to use online CL in object detection.”
• A Multi-Head Model for Continual Learning via Out-of-Distribution Replay (2022, methodology): “PCL (Hu et al., 2021) constructs an individual classifier for each class by using a fixed pre-trained feature extractor.”