Enabling Deep Spiking Neural Networks with Hybrid Conversion and Spike Timing Dependent Backpropagation

By Nitin Rathi, Gopalakrishnan Srinivasan, Priyadarshini Panda, Kaushik Roy. International Conference on Learning Representations, 2020

356 citations recorded by the source corpus.

Spiking Neural Networks (SNNs) operate with asynchronous discrete events (or spikes) which can potentially lead to higher energy-efficiency in neuromorphic hardware implementations. Many works have shown that an SNN for inference can be formed by copying the weights from a trained Artificial Neural Network (ANN) and setting the firing threshold for each layer as the maximum input received in that layer. These type of converted SNNs require a large number of time-steps to achieve competitive accuracy which diminishes the energy savings. The number of time-steps can be reduced by training SNNs with spike-based backpropagation from scratch, but that is computationally expensive and slow. To address these challenges, we present a computationally-efficient training technique for deep SNNs. We propose a hybrid training methodology: 1) take a converted SNN and use its weights and thresholds as an initialization step for spike-based backpropagation, and 2) perform incremental spike-timing dependent backpropagation (STDB) on this carefully initialized network to obtain an SNN that converges within few epochs and requires fewer time-steps for input processing. STDB is performed with a novel surrogate gradient function defined using neuron's spike time. The weight update is proportional to the difference in spike timing between the current time-step and the most recent time-step the neuron generated an output spike. The SNNs trained with our hybrid conversion-and-STDB training perform at 10X-25X fewer number of time-steps and achieve similar accuracy compared to purely converted SNNs. The proposed training methodology converges in less than 20 epochs of spike-based backpropagation for most standard image classification datasets, thereby greatly reducing the training complexity compared to training SNNs from scratch. We perform experiments on CIFAR-10, CIFAR-100 and ImageNet datasets for both VGG and ResNet architectures. We achieve top-1 accuracy of 65.19% for ImageNet dataset on SNN with 250 time-steps, which is 10X faster compared to converted SNNs with similar accuracy.

Subfields: Cognitive Neuroscience, Electrical and Electronic Engineering

View Paper: https://www.semanticscholar.org/paper/6dca2ee4cda6c75ba18e5efbcf8b1930c7c04ed8

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Electrical and Electronic Engineering

Citations in the corpus

Advancing Neuromorphic Computing With Loihi: A Survey of Results and Outlook (2021, methodology): “This is a well-understood and fundamental property of rate-coded feedforward networks [21], [31], [32].”
Spike-driven Transformer (2023, methodology): “Methods Architecture Spike -driven Param (M) Power (mJ) Time Step Acc Hybrid training [82] ResNet-34 3 21.”
Deep Residual Learning in Spiking Neural Networks (2021, methodology): “Spiking ResNet [25, 60, 21, 17, 49, 12, 30, 64, 48, 42, 43], as the spiking version of ResNet, is proposed by mimicking the residual block in ANNs and replacing ReLU activation layers with spiking neurons.”
DIET-SNN: A Low-Latency Spiking Neural Network With Direct Input Encoding and Leakage and Threshold Optimization (2021, background): “In recent work, the authors proposed a hybrid mechanism to circumvent the high training costs of backpropagation as well as maintain low inference latency (100–250 timesteps) [23].”
Incorporating Learnable Membrane Time Constant to Enhance Learning of Spiking Neural Networks (2020, methodology): “Most previous studies [51, 8, 48] preferred to use the average-pooling in SNNs as they found that max-pooling in SNNs leads to significant information loss.”