Biologically Inspired Sleep Algorithm for Reducing Catastrophic Forgetting in Neural Networks

By Timothy Tadros, Giri Panamoottil Krishnan, Ramyaa Ramyaa, Maxim Bazhenov. 2020

7 citations recorded by the source corpus.

Artificial neural networks (ANNs) are known to suffer from catastrophic forgetting: when learning multiple tasks, they perform well on the most recently learned task while failing to perform on previously learned tasks. In biological networks, sleep is known to play a role in memory consolidation and incremental learning. Motivated by the processes that are known to be involved in sleep generation in biological networks, we developed an algorithm that implements a sleep-like phase in ANNs. In an incremental learning framework, we demonstrate that sleep is able to recover older tasks that were otherwise forgotten. We show that sleep creates unique representations of each class of inputs and neurons that were relevant to previous tasks fire during sleep, simulating replay of previously learned memories.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Biological underpinnings for lifelong learning machines (2022, background): “Building on biological insights related to sleep and replay, it has recently been shown that both mimicking sleep 175–178 and adding internally generated replay 54,179 or rehearsal of stored data 180 , can help make deep neural networks more resistant to cata-strophic forgetting.”