Training Deep Spiking Neural Networks Using Backpropagation

By Junhaeng Lee, Tobi Delbruck, Michael Pfeiffer. Frontiers in Neuroscience, 2016

1,062 citations recorded by the source corpus.

Deep spiking neural networks (SNNs) hold the potential for improving the latency and energy efficiency of deep neural networks through data-driven event-based computation. However, training such networks is difficult due to the non-differentiable nature of spike events. In this paper, we introduce a novel technique, which treats the membrane potentials of spiking neurons as differentiable signals, where discontinuities at spike times are considered as noise. This enables an error backpropagation mechanism for deep SNNs that follows the same principles as in conventional deep networks, but works directly on spike signals and membrane potentials. Compared with previous methods relying on indirect training and conversion, our technique has the potential to capture the statistics of spikes more precisely. We evaluate the proposed framework on artificially generated events from the original MNIST handwritten digit benchmark, and also on the N-MNIST benchmark recorded with an event-based dynamic vision sensor, in which the proposed method reduces the error rate by a factor of more than three compared to the best previous SNN, and also achieves a higher accuracy than a conventional convolutional neural network (CNN) trained and tested on the same data. We demonstrate in the context of the MNIST task that thanks to their event-driven operation, deep SNNs (both fully connected and convolutional) trained with our method achieve accuracy equivalent with conventional neural networks. In the N-MNIST example, equivalent accuracy is achieved with about five times fewer computational operations.

Subfields: Cellular and Molecular Neuroscience, Cognitive Neuroscience, Electrical and Electronic Engineering

View Paper: https://www.semanticscholar.org/paper/c99537a3f0ce446df31942b78fb8337edb0c0c07

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cellular and Molecular Neuroscience

In Cognitive Neuroscience

In Electrical and Electronic Engineering

Citations in the corpus

Spiking-YOLO: Spiking Neural Network for Energy-Efficient Object Detection (2019, methodology): “Despite their excellent potential, SNNs have been limited to relatively simple tasks (e.g., image classiﬁcation) and small datasets (e.g., MNIST and CIFAR), on a rather shallow structure (Lee, Delbruck, and Pfeiffer 2016; Wu et al. 2019).”
Event-Based Vision: A Survey (2019, methodology): “This research is complemented by approaches where more computationally inspired types of supervised learning, such as back-propagation, are used in deep networks to efﬁciently implement spiking deep convolutional networks [127], [154], [155], [156], [157].”
Enabling Spike-Based Backpropagation for Training Deep Neural Network Architectures (2019, methodology): “This function is continuous except for the time points where spikes occur (Lee et al., 2016).”
Going Deeper With Directly-Trained Larger Spiking Neural Networks (2020, methodology): “Many methods have been proposed to deal with it, such as threshold regularization (Lee, Delbruck, and Pfeiffer 2016) and NeuNorm (Wu et al. 2019).”
Hebbian Learning based Orthogonal Projection for Continual Learning of Spiking Neural Networks (2024, background): “…by lateral signals, i.e., neuronal traces will consider both spike responses for feedforward inputs and responses for lateral circuits. et al., 2016; Zhang & Li, 2019; Zhou et al., 2021; Wu et al., 2021; Xiao et al., 2021; 2023) or consider the first spiking time as the encoding and derive…”
HATS: Histograms of Averaged Time Surfaces for Robust Event-Based Object Classification (2018, methodology): “The baselines methods we consider are HOTS [30], HFirst [50] and Spiking Neural Networks (SNN) [33, 46].”
Incorporating Learnable Membrane Time Constant to Enhance Learning of Spiking Neural Networks (2020, methodology): “Recently, the surrogate gradient method was proposed and provided another solution to training multi-layer SNNs [35, 26, 64, 57, 54, 34, 27].”
Spiking Neural Networks and Their Applications: A Review (2022, methodology): “Event-based methods, including SpikeProp, have the derivative term deﬁned only around the ﬁring time, whereas [45–47] ignore the temporal effect of the spike signal.”
RMP-SNN: Residual Membrane Potential Neuron for Enabling Deeper High-Accuracy and Low-Latency Spiking Neural Network (2020, methodology): “In order to scale the networks much deeper, spike-based error backpropagation algorithms have been proposed for the supervised training of SNNs [1, 15, 19, 21, 27, 29, 35, 41].”
Deep Learning With Spiking Neurons: Opportunities and Challenges (2018, methodology): “Recently, Lee et al. (2016) have introduced a spike-based backpropagation rule that can train deep SNNs for conventional classiﬁcation tasks (given only labels, but not target spike trains) directly from spike signals.”
Training Spiking Neural Networks Using Lessons From Deep Learning (2021, background): “Lee, Delbruck and Pfeiffer subsequently provided the first demonstration of approximating the backward path of spike signals in 2016 [178].”
Machine Learning for Wireless Networks with Artificial Intelligence: A Tutorial on Neural Networks (2017, background): “In addition, DNNs and CNNs can be used for modulation classification due to their capability of storing and extracting information from the data.”
Temporal Efficient Training of Spiking Neural Network via Gradient Re-weighting (2022, methodology): “They adopt surrogate gradients (SG) to relieve the non-differentiability (Lee et al., 2016; Wu et al., 2018; Zheng et al., 2021).”
SLAYER: Spike Layer Error Reassignment in Time (2018, methodology): “[13] with a comparable number of neurons.”