Spatio-Temporal Backpropagation for Training High-Performance Spiking Neural Networks

By Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, Luping P. Shi. Frontiers in Neuroscience, 2017

1,368 citations recorded by the source corpus.

Spiking neural networks (SNNs) are promising in ascertaining brain-like behaviors since spikes are capable of encoding spatio-temporal information. Recent schemes, e.g., pre-training from artificial neural networks (ANNs) or direct training based on backpropagation (BP), make the high-performance supervised training of SNNs possible. However, these methods primarily fasten more attention on its spatial domain information, and the dynamics in temporal domain are attached less significance. Consequently, this might lead to the performance bottleneck, and scores of training techniques shall be additionally required. Another underlying problem is that the spike activity is naturally non-differentiable, raising more difficulties in supervised training of SNNs. In this paper, we propose a spatio-temporal backpropagation (STBP) algorithm for training high-performance SNNs. In order to solve the non-differentiable problem of SNNs, an approximated derivative for spike activity is proposed, being appropriate for gradient descent training. The STBP algorithm combines the layer-by-layer spatial domain (SD) and the timing-dependent temporal domain (TD), and does not require any additional complicated skill. We evaluate this method through adopting both the fully connected and convolutional architecture on the static MNIST dataset, a custom object detection dataset, and the dynamic N-MNIST dataset. Results bespeak that our approach achieves the best accuracy compared with existing state-of-the-art algorithms on spiking networks. This work provides a new perspective to investigate the high-performance SNNs for future brain-like computing paradigm with rich spatio-temporal dynamics.

Subfields: Cognitive Neuroscience, Electrical and Electronic Engineering

View Paper: https://www.semanticscholar.org/paper/1d1802480eeb320d2f48bccee669c02804dbaa99

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Electrical and Electronic Engineering

Citations in the corpus

Incorporating Learnable Membrane Time Constant to Enhance Learning of Spiking Neural Networks (2020, methodology): “Recently, the surrogate gradient method was proposed and provided another solution to training multi-layer SNNs [35, 26, 64, 57, 54, 34, 27].”
Spikformer: When Spiking Neural Network Meets Transformer (2022, positive): “Spiking neurons receive continuous values and convert them into spike sequences, including the Leaky Integrate-and-Fire (LIF) neuron (Wu et al., 2018), PLIF (Fang et al., 2021b), etc.”
Advancing Neuromorphic Computing With Loihi: A Survey of Results and Outlook (2021, methodology): “…SNNs benchmarked on Loihi to date have used the Spike Layer Error Reassign-ment (SLAYER) algorithm [22], spatiotemporal backpropagation (STDB) [37], and a combination of BPTT with deep rewiring proposed in [36], which trains recurrent long short-term SNNs (LSNNs) with capabilities similar to…”
Spike-driven Transformer (2023, methodology): “The dynamics of the LIF layer [36] is governed by where t denote the timestep, U [ t ] means the membrane potential which is produced by coupling the spatial input information X [ t ] and the temporal input where X [ t ] can be obtained through operators such as Conv, MLP, and self-attention.”
Attention Spiking Neural Networks (2022, methodology): “Conv-Based LIF-SNN: Solving this differential equation, a simple iterative representation of LIF-SNN layer [36], [46] for easy inference and training is governed by ⎨⎩ U t,n = Ht−1,n +X S = Hea ( U t,n − uth )”
Deep Residual Learning in Spiking Neural Networks (2021, methodology): “The method in the first category computes the gradient by unfolding the network over the simulation timesteps [31, 19, 58, 50, 30, 40], which is similar to the idea of backpropagation through time (BPTT).”
Temporal Efficient Training of Spiking Neural Network via Gradient Re-weighting (2022, methodology): “Following the concept of direct training, we regard the SNN as RNN and calculate the gradients through spatial-temporal backpropagation (STBP) (Wu et al., 2018): where the term ∂ a ( t ) ∂ u ( t ) is the gradient of the non-differentiability step function involving the derivative of Dirac’s δ…”
Hebbian Learning based Orthogonal Projection for Continual Learning of Spiking Neural Networks (2024, methodology): “…are calculated as: where ∂L ∂ s l +1 [ t ] is the error backpropagated to layer l + 1 at time t , and the non-differentiable terms ∂ u l [ t ] will be replaced by surrogate derivatives, e.g., derivatives of rectangular or sigmoid functions (Wu et al., 2018): ) 2 ( a 1 and a 2 are hyperparameters).”
Enabling Spike-Based Backpropagation for Training Deep Neural Network Architectures (2019, background): “…are “Spike Time Dependent Plasticity (STDP)” based unsupervised learning (Brader et al., 2007; Diehl and Cook, 2015; Srinivasan et al., 2018a,b) and “spike-based backpropagation” based supervised learning (Lee et al., 2016; Neftci et al., 2017; Jin et al., 2018; Mostafa, 2018; Wu et al., 2018b).”
Enabling Deep Spiking Neural Networks with Hybrid Conversion and Spike Timing Dependent Backpropagation (2020, methodology): “…of the spiking neuron is handled by either approximating the spiking neuron model as continuous and differentiable (Huh & Sejnowski, 2018) or by deﬁning a surrogate gradient as a continuous approximation of the real gradient (Wu et al., 2018; Bellec et al., 2018; Neftci et al., 2019).”
Direct Training for Spiking Neural Networks: Faster, Larger, Better (2018, methodology): “Based on this analysis, we integrate our modified LIF model, coding scheme, and proposed NeuNorm into the STBP method (Wu et al. 2018) to train our network.”
Event-Based Vision: A Survey (2019, background): “achieving over 98 % accuracy on the task [107], [138], [139], [234], [235], [236], but few would propose digit recognition”
SpikingJelly: An open-source machine learning infrastructure platform for spike-based intelligence (2023, methodology): “Frame representations are widely used ( 45 , 56 , 57 , 60 , 98 , 99 ) and are usually temporally down-sampled from event representations.”