CoSCL: Cooperation of Small Continual Learners is Stronger than a Big One

By Liyuan Wang, Xingxing Zhang, Qian Li, Jun Zhu, Yi Zhong. 2022

78 citations recorded by the source corpus.

Continual learning requires incremental compatibility with a sequence of tasks. However, the design of model architecture remains an open question: In general, learning all tasks with a shared set of parameters suffers from severe interference between tasks; while learning each task with a dedicated parameter subspace is limited by scalability. In this work, we theoretically analyze the generalization errors for learning plasticity and memory stability in continual learning, which can be uniformly upper-bounded by (1) discrepancy between task distributions, (2) flatness of loss landscape and (3) cover of parameter space. Then, inspired by the robust biological learning system that processes sequential experiences with multiple parallel compartments, we propose Cooperation of Small Continual Learners (CoSCL) as a general strategy for continual learning. Specifically, we present an architecture with a fixed number of narrower sub-networks to learn all incremental tasks in parallel, which can naturally reduce the two errors through improving the three components of the upper bound. To strengthen this advantage, we encourage to cooperate these sub-networks by penalizing the difference of predictions made by their feature representations. With a fixed parameter budget, CoSCL can improve a variety of representative continual learning approaches by a large margin (e.g., up to 10.64% on CIFAR-100-SC, 9.33% on CIFAR-100-RS, 11.45% on CUB-200-2011 and 6.72% on Tiny-ImageNet) and achieve the new state-of-the-art performance.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Hang Su
2. Jun Zhu
3. Liyuan Wang
4. Xingxing Zhang
5. Liyuan Wang
6. Hongwei Yan
7. Kaisheng Ma
8. Yi Zhong

In Computer Vision and Pattern Recognition

1. Hang Su
2. Jun Zhu
3. Liyuan Wang
4. Xingxing Zhang
5. Liyuan Wang
6. Hongwei Yan
7. Kaisheng Ma
8. Yi Zhong

In Meta & Continual Learning

1. Liyuan Wang
2. Xingxing Zhang
3. Hang Su
4. Liyuan Wang
5. Hongwei Yan
6. Kaisheng Ma
7. Yi Zhong
8. Jun Zhu

Citations in the corpus

• A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, methodology): “Of note, the above analyses are mainly based on finding a shared solution for all tasks, which is subject to severe inter-task interference [52], [68], [69].”
• Incorporating neuro-inspired adaptability for continual learning in artificial intelligence (2023, methodology): “The discrepancy of task distributions in feature space is evaluated by the difficulty of distinguishing them, which is an empirical approximation of the H -divergence [47, 53] in Proposition 1.”
• Divide and not forget: Ensemble of selectively trained experts in Continual Learning (2024, methodology): “…extractor is trained on the larger first task, which provides a substantial amount of data (i.e., 50% of all available classes) (Hou et al., 2019; Zhu et al., 2022; Petit et al., 2023), or it starts from a large pre-trained model that remains unchanged (Hayes & Kanan, 2020a; Wang et al., 2022c)…”
• Orchestrate Latent Expertise: Advancing Online Continual Learning with Multi-Level Supervision and Reverse Self-Distillation (2024, methodology): “Architecture-based methods focus on allocating dedicated parameter subspace, including parameter isolation [50, 68], dynamic architecture [17, 46, 59, 65], and modular network [62, 64].”