Enabling Spike-Based Backpropagation for Training Deep Neural Network Architectures

By Chankyu Lee, Syed Shakib Sarwar, Priyadarshini Panda, Gopalakrishnan Srinivasan, Kaushik Roy. Frontiers in Neuroscience, 2019

452 citations recorded by the source corpus.

Spiking Neural Networks (SNNs) have recently emerged as a prominent neural computing paradigm. However, the typical shallow SNN architectures have limited capacity for expressing complex representations while training deep SNNs using input spikes has not been successful so far. Diverse methods have been proposed to get around this issue such as converting off-the-shelf trained deep Artificial Neural Networks (ANNs) to SNNs. However, the ANN-SNN conversion scheme fails to capture the temporal dynamics of a spiking system. On the other hand, it is still a difficult problem to directly train deep SNNs using input spike events due to the discontinuous, non-differentiable nature of the spike generation function. To overcome this problem, we propose an approximate derivative method that accounts for the leaky behavior of LIF neurons. This method enables training deep convolutional SNNs directly (with input spike events) using spike-based backpropagation. Our experiments show the effectiveness of the proposed spike-based learning on deep networks (VGG and Residual architectures) by achieving the best classification accuracies in MNIST, SVHN, and CIFAR-10 datasets compared to other SNNs trained with a spike-based learning. Moreover, we analyze sparse event-based computations to demonstrate the efficacy of the proposed SNN training method for inference operation in the spiking domain.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Electrical and Electronic Engineering

View Paper: https://www.semanticscholar.org/paper/eaaaed86d1b811fb4690e20ec532d4298c10e324

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Electrical and Electronic Engineering

Citations in the corpus

DIET-SNN: A Low-Latency Spiking Neural Network With Direct Input Encoding and Leakage and Threshold Optimization (2021, methodology): “There have been many proposals to perform gradient-based training in SNNs [18], [25]—among them, the most successful is surrogate-gradient-based optimization [12].”
Deep Residual Learning in Spiking Neural Networks (2021, methodology): “The method in the first category computes the gradient by unfolding the network over the simulation timesteps [31, 19, 58, 50, 30, 40], which is similar to the idea of backpropagation through time (BPTT).”
Going Deeper With Directly-Trained Larger Spiking Neural Networks (2020, methodology): “With gradient descent optimization learning algorithms, some SNN models (Lee, Delbruck, and Pfeiffer 2016; Jin, Zhang, and Li 2018; Lee et al. 2020) achieve high performance on CIFAR-10 and other neuro-morphic datasets.”
Incorporating Learnable Membrane Time Constant to Enhance Learning of Spiking Neural Networks (2020, methodology): “Recently, the surrogate gradient method was proposed and provided another solution to training multi-layer SNNs [35, 26, 64, 57, 54, 34, 27].”
Spikformer: When Spiking Neural Network Meets Transformer (2022, methodology): “Because the event-triggered mechanism in spiking neurons is non-differentiable, the surrogate gradient is used for backpropagation (Lee et al., 2020; Neftci et al., 2019)Xiao et al. (2021a) adopts implicit differentiation on the equilibrium state to train SNN.”
Opportunities for neuromorphic computing algorithms and applications (2022, background): “There have also been a few demonstrations on computing the spike error gradient 37–39 that have shown close to state-of-the-art classification performance on the Modified National Institute of Standards and Technology (MNIST) handwritten digits dataset.”