Deep Learning in Spiking Neural Networks

By Amirhossein Tavanaei, Masoud Ghodrati, S. R. Kheradpisheh, Timothée Masquelier, Anthony S. Maida. Neural Networks, 2018

1,376 citations recorded by the source corpus.

In recent years, deep learning has revolutionized the field of machine learning, for computer vision in particular. In this approach, a deep (multilayer) artificial neural network (ANN) is trained, most often in a supervised manner using backpropagation. Vast amounts of labeled training examples are required, but the resulting classification accuracy is truly impressive, sometimes outperforming humans. Neurons in an ANN are characterized by a single, static, continuous-valued activation. Yet biological neurons use discrete spikes to compute and transmit information, and the spike times, in addition to the spike rates, matter. Spiking neural networks (SNNs) are thus more biologically realistic than ANNs, and are arguably the only viable option if one wants to understand how the brain computes at the neuronal description level. The spikes of biological neurons are sparse in time and space, and event-driven. Combined with bio-plausible local learning rules, this makes it easier to build low-power, neuromorphic hardware for SNNs. However, training deep SNNs remains a challenge. Spiking neurons' transfer function is usually non-differentiable, which prevents using backpropagation. Here we review recent supervised and unsupervised methods to train deep SNNs, and compare them in terms of accuracy and computational cost. The emerging picture is that SNNs still lag behind ANNs in terms of accuracy, but the gap is decreasing, and can even vanish on some tasks, while SNNs typically require many fewer operations and are the better candidates to process spatio-temporal data.

Subfields: Cellular and Molecular Neuroscience, Cognitive Neuroscience, Electrical and Electronic Engineering

View Paper: https://www.semanticscholar.org/paper/1a13bec308614ed56b9af2532a3d5e4cd8e2af94

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cellular and Molecular Neuroscience

In Cognitive Neuroscience

In Electrical and Electronic Engineering

Citations in the corpus

Direct Training for Spiking Neural Networks: Faster, Larger, Better (2018, background): “STDP is biologically plausible but the lack of global information hinders the convergence of large models, especially on complex datasets (Timothe and Thorpe 2007; Diehl and Matthew 2015; Tavanaei and Maida 2017).”
Automated medical diagnosis of COVID-19 through EfficientNet convolutional neural network (2020, background): “The hidden layers usually consist of convolutional layers, ReLU layers, pooling layers, and fully connected layers [35,36].”
Artificial Neural Networks for Neuroscientists: A Primer (2020, methodology): “These methods generally involve approximating spiking networks with a differentiable system [Tavanaei et al., 2019].”
Learning to Detect Objects with a 1 Megapixel Event Camera (2020, methodology): “Some of these methods, such as Spiking Neural Networks [14, 15, 16, 17], exploit the sparsity of the data and can be applied event by event, to preserve the temporal resolution of the events stream [15, 18, 19, 20].”
EEG based multi-class seizure type classification using convolutional neural network and transfer learning (2020, methodology): “A review was given on recent supervised and unsupervised methods to train deep spiking neural networks and compared them in terms of accuracy and computational cost (Tavanaei, Ghodrati, Kheradpisheh, Masquelier, & Maida, 2018).”
SpikingJelly: An open-source machine learning infrastructure platform for spike-based intelligence (2023, methodology): “Deeplearning methods boost the performance of SNNs and make SNNs practical for use in real-life tasks ( 26 ).”
A review of learning in biologically plausible spiking neural networks (2019, neutral): “Tavanaei et al. (Tavanaei et al., 2019) compared a number of DSNNs, and their result shows that deep SNNs which are converted from a classical DNN can achieve the highest accuracy on MNIST data compared to methods that train DSNNs directly.”
The Heidelberg Spiking Data Sets for the Systematic Evaluation of Spiking Neural Networks (2019, methodology): “Furthermore, there are biologically inspired online approximations of surrogate gradients [2], [73], [74] and, finally, a body of work has used biologically motivated spiketiming dependent plasticity (STDP)-like learning rules [75], [76] (see [4] for a comprehensive review).”