Improved multitask learning through synaptic intelligence

By Friedemann Zenke, Ben Poole, Surya Ganguli. 2017

55 citations recorded by the source corpus.

Deep learning has led to remarkable advances when applied to problems where the data distribution does not change over the course of learning. In stark contrast, biological neural networks continually adapt to changing domains, and solve a diversity of tasks simultaneously. Furthermore, synapses in biological neurons are not simply real-valued scalars, but possess complex molecular machinery enabling non-trivial learning dynamics. In this study, we take a first step toward bringing this biological complexity into artificial neural networks. We introduce a model of intelligent synapses that accumulate task relevant information over time, and exploit this information to efficiently consolidate memories of old tasks to protect them from being overwritten as new tasks are learned. We apply our framework to learning sequences of related classification problems, and show that it dramatically reduces catastrophic forgetting while maintaining computational efficiency.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Meta & Continual Learning, Neuroscience

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

In Meta & Continual Learning

In Neuroscience

Citations in the corpus

Overcoming catastrophic forgetting with hard attention to the task (2018, methodology): “Both elastic weight consolidation (EWC; Kirkpatrick et al., 2017) and synaptic intelligence (SI; Zenke et al., 2017) approaches add a soft structural regularization term to the loss function in order to discourage changes to weights that are important for previous tasks.”
Gradient Episodic Memory for Continual Learning (2017, methodology): “Within this class of methods, there are approaches that leverage “synaptic” memory [Kirkpatrick et al., 2017, Zenke et al., 2017], where learning rates are adjusted to minimize changes in parameters important for previous tasks.”
Probing Representation Forgetting in Supervised and Unsupervised Continual Learning (2022, methodology): “The second category of methods encode the knowledge of the previous tasks in a prior that is used to regularize the training of the new task [1, 22, 26, 34, 49].”
Online Continual Learning with Maximally Interfered Retrieval (2019, methodology): “Under the fixed architecture setting, one can identify two main streams of works: i) methods that rely on replaying samples or virtual (generated) samples from the previous history while learning new ones and ii) methods that encode the knowledge of the previous tasks in a prior that is used to regu”
Learning to Remember: A Synaptic Plasticity Driven Framework for Continual Learning (2019, background): “Lately, a wealth of works dealing with catastrophic forgetting in context of DNNs have appeared in the literature, see e.g., [9, 34, 12, 29, 1, 24].”
Task-Free Continual Learning (2018, methodology): “Task-based sequential learning has mostly been studied for image classiﬁcation [19, 2, 17, 39, 28].”
Memory Aware Synapses: Learning what (not) to forget (2017, methodology): “Before introducing our method, we briefly remind the reader of the standard LLL setup, and summarize the work on Synaptic Intelligence [31], on which we build.”
Continual Learning in Generative Adversarial Nets (2017, methodology): “A similar method is proposed in [19].”
Generative replay with feedback connections as a general strategy for continual learning (2018, methodology): “To obscure things further, some methods shown to perform well in some experimental settings are reported to dramatically fail in others ( e.g. , compare the performance of elastic weight consolidation in [1] and [7] with that in [8] and [9]).”
Selfless Sequential Learning (2018, methodology): “A similar setup has been used in [16, 47, 13] with different percentage of permutations or different number of tasks.”
Revisiting Distillation and Incremental Classifier Learning (2018, neutral): “weights when learning new tasks [17] [18].”
Overcoming Catastrophic Forgetting with Hard Attention to the Task (2018, methodology): “To match the different number of parameters used in the literature, we consider a small, medium, and a large network based on a two-layer fully-connected architecture as Zenke et al. (2017), with 100, 500, and 2000 hidden units, respectively.”
CLRS: Continual Learning Benchmark for Remote Sensing Image Scene Classification (2020, methodology): “For example, Friedemann et al. [15] split the full MNIST [21] training dataset into five batches, each of which contains two classes.”
Memory Aware Synapses: Learning what (not) to forget (2018, methodology): “In the Synaptic Intelligence work [39], importance weights are computed during training in an online manner.”
Continual Learning in Neural Networks (2019, methodology): “Like other prior-focused methods [69, 168], we estimate an importance weight for each parameter in the network.”
Self-Net: Lifelong Learning via Continual Self-Modeling (2018, methodology): “Existing methods, though, have at least one of three limitations: they either (1) restrict the network’s ability to learn new tasks by penalizing changes to existing weights [7, 27, 22, 14]; (2) expand the model size linearly as the number of tasks grows [21, 10] (or dynamically define task-specific”
Efficient Continual Learning in Neural Networks with Embedding Regularization (2019, background): “WF has the advantage that the training process can potentially discover new weight congurations able to preserve older behaviors. 2.4. Synaptic Intelligence Synaptic Intelligence (SI), introduced in [32], is a variant of EWC. The main idea is to calculate the weight importance in an online fashion, ”