Sleep prevents catastrophic forgetting in spiking neural networks by forming a joint synaptic weight representation

By Ryan Golden, Jean Erik Delanois, Pavel Sanda, Maxim Bazhenov. 2022

30 citations recorded by the source corpus.

Artificial neural networks overwrite previously learned tasks when trained sequentially, a phenomenon known as catastrophic forgetting. In contrast, the brain learns continuously, and typically learns best when new training is interleaved with periods of sleep for memory consolidation. Here we used spiking network to study mechanisms behind catastrophic forgetting and the role of sleep in preventing it. The network could be trained to learn a complex foraging task but exhibited catastrophic forgetting when trained sequentially on different tasks. In synaptic weight space, new task training moved the synaptic weight configuration away from the manifold representing old task leading to forgetting. Interleaving new task training with periods of off-line reactivation, mimicking biological sleep, mitigated catastrophic forgetting by constraining the network synaptic weight state to the previously learned manifold, while allowing the weight configuration to converge towards the intersection of the manifolds representing old and new tasks. The study reveals a possible strategy of synaptic weights dynamics the brain applies during sleep to prevent forgetting and optimize learning.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Tyler L. Hayes
2. Christopher Kanan
3. Maxim Bazhenov
4. Giri Panamoottil Krishnan
5. Hava T. Siegelmann
6. Terrence J. Sejnowski
7. Vincenzo Lomonaco
8. Kushal Kafle

In Cognitive Neuroscience

1. Maxim Bazhenov
2. Giri Panamoottil Krishnan
3. Tyler L. Hayes
4. Terrence J. Sejnowski
5. Hava T. Siegelmann
6. Christopher Kanan
7. Yoshua Bengio
8. Laurenz Wiskott

In Electrical and Electronic Engineering

1. Giri Panamoottil Krishnan
2. Maxim Bazhenov
3. Terrence J. Sejnowski
4. Tyler L. Hayes
5. Christopher Kanan

Citations in the corpus

• Replay in Deep Learning: Current Approaches and Missing Biological Elements (2021, positive): “The studies demonstrated that replay helps to avoid interference between competing memories and results in a synaptic connection that reflects the combination of previous tasks (Golden, Delanois, Sanda, & Bazhenov, 2020; González et al., 2020).”