FearNet: Brain-Inspired Model for Incremental Learning

By Ronald Kemker, Christopher Kanan. 2017

523 citations recorded by the source corpus.

Incremental class learning involves sequentially learning classes in bursts of examples from the same class. This violates the assumptions that underlie methods for training standard deep neural networks, and will cause them to suffer from catastrophic forgetting. Arguably, the best method for incremental class learning is iCaRL, but it requires storing training examples for each class, making it challenging to scale. Here, we propose FearNet for incremental class learning. FearNet is a generative model that does not store previous examples, making it memory efficient. FearNet uses a brain-inspired dual-memory system in which new memories are consolidated from a network for recent memories inspired by the mammalian hippocampal complex to a network for long-term storage inspired by medial prefrontal cortex. Memory consolidation is inspired by mechanisms that occur during sleep. FearNet also uses a module inspired by the basolateral amygdala for determining which memory system to use for recall. FearNet achieves state-of-the-art performance at incremental class learning on image (CIFAR-100, CUB-200) and audio classification (AudioSet) benchmarks.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Meta & Continual Learning, Neuroscience

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Meta & Continual Learning

In Neuroscience

Citations in the corpus

Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselines (2018, background): “) and use one of them in one task [17, 18, 19, 20].”
Continual Lifelong Learning with Neural Networks: A Review (2018, background): “ low complexity, e.g. the MNIST and the Street View House Number (SVHN) (Netzer et al., 2014). Consequently, the question arises whether this generative approach can scale up to more complex domains. Kemker and Kanan (2018) proposed the FearNet model for incremental class learning that is inspired b”
A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, methodology): “In contrast, autoencoder-based strategies, such as FearNet [210], SRM [361], CLEER [366], EEC [26], GMR [334] and Flashcards [138], can explicitly control the labels of the generated data, although relatively blurred.”
On the role of neurogenesis in overcoming catastrophic forgetting (2018, background): “Proposed solutions for mitigating catastrophic interference in this domain have focused on regulating intrinsic levels of plasticity to protect acquired knowledge given a fixed number of neurons [5, 6, 13], dynamically allocating new neurons to accommodate novel dissimilar knowledge [7, 8, 16] and s”
Continuous Learning in Single-Incremental-Task Scenarios (2018, methodology): “In particular, FearNet (FN) [11] and Growing Dual-Memory (GDM) [26] are interesting approaches leveraging ideas from architectural and (pseudo) rehearsal categories: a double-memory system is exploited to learn new concepts in a short-term memory and progressively consolidate them in a long-term one”
Lifelong Learning of Spatiotemporal Representations With Dual-Memory Recurrent Self-Organization (2018, background): “Kemker and Kanan (2018) proposed the FearNet model for incremental class learning inspired by studies of memory recall and consolidation in the mammalian brain during fear conditioning (Kitamura et al., 2017).”
Closed-Loop GAN for continual Learning (2018, neutral): “Regularization methods have been shown to perform poorly in incremental class learning [Kemker and Kanan, 2017, Parisi et al., 2018] and here we reproduce this limitation in our own results for the case of Elastic Weight Consolidation [Kirkpatrick et al., 2017].”
Memory Efficient Experience Replay for Streaming Learning (2018, background): “two networks [26], [40]: one that learns fast inspired by the hippocampus and one that learns slowly inspired by the neocortex [59].”
Learnability and Algorithm for Continual Learning (2023, methodology): “Pseudoreplay methods generate pseudo replay data for old tasks to serve as the replay data (Kamra et al., 2017; Shin et al., 2017; Wu et al., 2018; Seff et al., 2017; Kemker & Kanan, 2018; Hu et al., 2019; Rostami et al., 2019; Ostapenko et al., 2019).”
CODA-Prompt: COntinual Decomposed Attention-Based Prompting for Rehearsal-Free Continual Learning (2022, background): “Rehearsal with stored data [3–5, 7, 9, 10, 18, 19, 24, 29, 42, 51–53, 64, 68] or samples from a generative model [27, 28,48,56,60] is highly effective when storing training data or training/saving a generative model is possible.”
A Theoretical Study on Solving Continual Learning (2022, background): “Some systems learn to generate pseudo training data of old tasks to be used in replay, called pseudo-replay [41, 42, 43, 44, 45, 46, 21, 47, 48].”
ALIFE: Adaptive Logit Regularizer and Feature Replay for Incremental Semantic Segmentation (2022, methodology): “To avoid these problems, recent works employ a pseudo replay scheme that synthesizes images [20, 31, 34] or latent features [38] of previous categories.”
S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learning (2022, methodology): “For instance, memory-based methods alleviate forgetting by either replaying a small set of examples from past tasks saved in a memory [57, 55, 46, 7, 23, 28, 9] or replaying synthesized data from previous tasks using generative models [60, 63].”
Adaptive Orthogonal Projection for Batch and Online Continual Learning (2022, background): “Many other papers use related approaches (Zenke, Poole, and Ganguli 2017; Fernando et al. 2017; Aljundi et al. 2017; Ritter, Botev, and Barber 2018; Xu and Zhu 2018; Kemker and Kanan 2018; Parisi et al. 2018; Ahn et al. 2019; Hu et al. 2021; Dhar et al. 2019; Adel, Zhao, and Turner 2020).”
vCLIMB: A Novel Video Class Incremental Learning Benchmark (2022, background): “Although there is extensive literature on continual learning [1, 7, 8, 15, 17, 21, 25, 27, 31, 42], we restrict our review to a sample of the most widely adopted baselines.”
Continual Learning of a Mixed Sequence of Similar and Dissimilar Tasks (2021, methodology): “Some methods memorize a small set of training examples in each task and use them in learning a new task to deal with forgetting (called replay ) (Rebufﬁ et al., 2017; Lopez-Paz and Ranzato, 2017; Chaudhry et al., 2019; Wu et al., 2018; Kemker and Kanan, 2018).”
DyTox: Transformers for Continual Learning with DYnamic TOken eXpansion (2021, methodology): “In the ﬁrst row, a different TAB is created per task, while in the second row the same TAB is used — which is the DyTox strategy.”
Overcoming Catastrophic Forgetting in Incremental Few-Shot Learning by Finding Flat Minima (2021, background): “Rooted from real-world applications, this topic has attracted a significant amount of interest in recent years [5, 30, 31, 40, 26].”
Bridging Non Co-occurrence with Unlabeled In-the-wild Data for Incremental Object Detection (2021, neutral): “They usually keep a small number of exemplars [22, 29, 3], or generate synthetic images [26, 19] or features [12, 30] to achieve this purpose.”