Continual Learning, Fast and Slow

By Quang Pham, Chenghao Liu, Steven C. H. Hoi. 2021

57 citations recorded by the source corpus.

According to the Complementary Learning Systems (CLS) theory (McClelland et al. 1995) in neuroscience, humans do effective continual learning through two complementary systems: a fast learning system centered on the hippocampus for rapid learning of the specifics, individual experiences; and a slow learning system located in the neocortex for the gradual acquisition of structured knowledge about the environment. Motivated by this theory, we propose DualNets (for Dual Networks), a general continual learning framework comprising a fast learning system for supervised learning of pattern-separated representation from specific tasks and a slow learning system for representation learning of task-agnostic general representation via Self-Supervised Learning (SSL). DualNets can seamlessly incorporate both representation types into a holistic framework to facilitate better continual learning in deep neural networks. Via extensive experiments, we demonstrate the promising results of DualNets on a wide range of continual learning protocols, ranging from the standard offline, task-aware setting to the challenging online, task-free scenario. Notably, on the CTrL (Veniat et al. 2020) benchmark that has unrelated tasks with vastly different visual images, DualNets can achieve competitive performance with existing state-of-the-art dynamic architecture strategies (Ostapenko et al. 2021). Furthermore, we conduct comprehensive ablation studies to validate DualNets efficacy, robustness, and scalability.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Online Continual Learning with Contrastive Vision Transformer (2022, methodology): “We consider a strict evaluation setting [38,75] for online continual learning, including task-aware protocol [56] and task-free protocol [39].”
DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning (2022, methodology): “We argue such a design could be sub-optimal from the perspective of theory of Complementary Learning Systems (CLS) [37, 22], an intuition that many recent advanced CL methods are based on [7, 43, 3].”
Consistency is the key to further mitigating catastrophic forgetting in continual learning (2022, background): “Therefore, several works (Benjamin et al., 2018; Buzzega et al., 2020; Arani et al., 2022; Pham et al., 2021) leverage soft-targets in addition to hard ones thereby drawing more information from the limited buffered samples.”
How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Composition (2022, methodology): “As another example, Pham et al. (2021b) used a dual memory to train a set of shared neural network layers and a task-specific controller to transform the features of the shared layers.”
Effects of Auxiliary Knowledge on Continual Learning (2022, background): “Other works prioritize the learning of high-level representations either by adopting learning objectives designed not to disrupt the performance on previous tasks [32], [33], [7], [34], or by making use of semi-supervised learning techniques to learn general features [35], [36], [37].”