Continual Learning Through Synaptic Intelligence

By Friedemann Zenke, Ben Poole, Surya Ganguli. 2017

3,378 citations recorded by the source corpus.

While deep learning has led to remarkable advances across diverse applications, it struggles in domains where the data distribution changes over the course of learning. In stark contrast, biological neural networks continually adapt to changing domains, possibly by leveraging complex molecular machinery to solve many tasks simultaneously. In this study, we introduce intelligent synapses that bring some of this biological complexity into artificial neural networks. Each synapse accumulates task relevant information over time, and exploits this information to rapidly store new memories without forgetting old ones. We evaluate our approach on continual learning of classification tasks, and show that it dramatically reduces forgetting while maintaining computational efficiency.

Subfields: Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

Citations in the corpus

Parameter-Efficient Transfer Learning for NLP (2019, background): “Techniques have been proposed to mitigate forgetting (Kirk- patrick et al., 2017; Zenke et al., 2017), however, unlike for adapters, the memory is still imperfect.”
Toward Understanding Catastrophic Forgetting in Continual Learning (2019, methodology): “Similar constructions can also be applied to the not-MNIST set [7], the fashion MNIST set [48], or the CIFAR set [23] such as in the split not-MNIST [29] and split CIFAR benchmarks [51].”
Self-Supervised GANs via Auxiliary Rotation Loss (2018, methodology): “…is to introduce Method Accuracy Context [24] 0.317 BiGAN [34, 36] 0.310 Colorization [25] 0.326 RotNet [26] 0.387 DeepClustering [35] [15] counter discriminator forgetting by leveraging techniques from continual learning directly (Elastic Weight Sharing [11] and Intelligent Synapses [37]).”
Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselines (2018, methodology): “For comparison, we pick several popular methods (EWC[3], online EWC[23], SI[9], MAS[24], LwF[25]) and state-of-the-art rehearsal-based methods (DGR[8], RtF[14]) with generative model.”
Fast and Flexible Multi-Task Classification Using Conditional Neural Adaptive Processes (2019, methodology): “We meta-test CNAPS on three additional held-out datasets: MNIST [29], CIFAR10 [30], and CIFAR100 [30].”
Continual Lifelong Learning with Neural Networks: A Review (2018, background): “rich set of neurophysiological processing principles and cognitive functions that together contribute to the early development and experience-driven specialization of our perceptual and motor skills (Zenke et al., 2017a; Power and Schlaggar, 2017; Murray et al., 2016; Lewkowicz, 2014). In Section 2,”
Experience Replay for Continual Learning (2018, methodology): “Outside of RL, Zenke et al. [32] develop a method similar to EWC that maintains estimates of the importance of weights for past tasks, Li and Hoiem [15] leverage a mixture of task-speciﬁc and shared parameters, and Milan et al. [19] develop a rigorous Bayesian approach for estimating unknown task…”
A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, methodology): “SI [105] online approximates the importance of each parameter by its contribution to the total loss variation and its update length over the entire training trajectory.”
Variational Continual Learning (2017, methodology): “Split MNIST: This experiment was used by Zenke et al. (2017) to assess the SI method.”
DisCoRL: Continual Reinforcement Learning via Policy Distillation (2019, methodology): “…of applying regularization , either by constraining weight updates in order to maintain knowledge from previous tasks [Kirkpatrick et al., 2017, Zenke et al., 2017, Maltoni and Lomonaco, 2018], or by keeping an old model in memory and distilling knowledge [Hinton et al., 2015] into it later to…”
Continual and Multi-Task Architecture Search (2019, methodology): “Others proposed copying the previous task and augmenting with new task’s features (Rusu et al., 2016), intelligent synapses to accumulate task-related information (Zenke et al., 2017), or online variational inference (Nguyen et al., 2017).”
On the role of neurogenesis in overcoming catastrophic forgetting (2018, background): “Proposed solutions for mitigating catastrophic interference in this domain have focused on regulating intrinsic levels of plasticity to protect acquired knowledge given a fixed number of neurons [5, 6, 13], dynamically allocating new neurons to accommodate novel dissimilar knowledge [7, 8, 16] and s”
Continuous Learning in Single-Incremental-Task Scenarios (2018, methodology): “-200. CORe50 is one of the few datasets speciﬁcally designed for CL in a SIT scenario. Dataset Multi-Task (MT) Single-Incremental-Task (SIT) Permuted MNIST [14][21] - Rotated MNIST [21] - MNIST Split [33] [25] CIFAR-10/100 Split [33] - iCIFAR-100 [21]? [28] ILSVRC2012 - [28] CUB-200 - [25] CORe50 - ”
Lifelong Learning of Spatiotemporal Representations With Dual-Memory Recurrent Self-Organization (2018, background): “Zenke et al. (2017b) proposed to alleviate catastrophic forgetting by allowing individual synapses to estimate their importance for solving a learned task.”
Critical Learning Periods in Deep Neural Networks (2017, background): “However, not all the connections have the same importance for the final output of the network: This fact is captured, for example, by the Fisher Information Matrix (FIM) (35), which may be considered the artificial network equivalent of population-level measures of synaptic strength and connectivity”
Deep Generative Dual Memory Network for Continual Learning (2017, methodology): “Our approach allows online recognition of task samples via a reconstructive generative model and is applicable in domains with directly streaming data without any task descriptors unlike most previous approaches which make explicit use of task descriptors (Zenke et al., 2017; Kirkpatrick et al., 201”
Lifelong Learning with Dynamically Expandable Networks (2017, neutral): “…is studied in deep learning frameworks; since lifelong learning of a deep network can be straightforwardly done by simple re-training, the primary focus of research is on overcoming catastrophic forgetting Kirkpatrick et al. (2017); Rusu et al. (2016); Zenke et al. (2017); Lee et al. (2017).”
Synaptic weight decay with selective consolidation enables fast learning without catastrophic forgetting (2019, background): “Forgetting can also be counterbalanced by reducing plasticity for weight conﬁgurations that are important in previous contexts, but assessing the importance of a synaptic weight requires additional information and memories (Kirkpatrick et al., 2017; Zenke et al., 2017; Aljundi et al., 2018).”
Policy Consolidation for Continual Reinforcement Learning (2019, methodology): “Approaches to mitigating catastrophic forgetting in neural networks include ones that selectively regularise the parameters to preserve knowledge from previously trained tasks (Kirkpatrick et al., 2017; Zenke et al., 2017; Li & Hoiem, 2017), ones that allocate more neural resources over time (Ring, ”
Closed-Loop GAN for continual Learning (2018, background): “Similarly, Zenke et al. [2017] use path integrals of loss-derivatives to constrain crucial weights, yielding an intermediate parameterization with minimal combined loss.”