SLAYER: Spike Layer Error Reassignment in Time

By Sumit Bam Shrestha, Garrick Orchard. Neural Information Processing Systems, 2018

931 citations recorded by the source corpus.

Configuring deep Spiking Neural Networks (SNNs) is an exciting research avenue for low power spike event based computation. However, the spike generation function is non-differentiable and therefore not directly compatible with the standard error backpropagation algorithm. In this paper, we introduce a new general backpropagation mechanism for learning synaptic weights and axonal delays which overcomes the problem of non-differentiability of the spike function and uses a temporal credit assignment policy for backpropagating error to preceding layers. We describe and release a GPU accelerated software implementation of our method which allows training both fully connected and convolutional neural network (CNN) architectures. Using our software, we compare our method against existing SNN based learning approaches and standard ANN to SNN conversion techniques and show that our method achieves state of the art performance for an SNN on the MNIST, NMNIST, DVS Gesture, and TIDIGITS datasets.

Subfields: Geophysics

View Paper: https://www.semanticscholar.org/paper/3fe019f5ee31a4be3fe24b687f555006e17d27c5

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Geophysics

Citations in the corpus

Spikformer: When Spiking Neural Network Meets Transformer (2022, methodology): “In the area of direct training, SNNs are unfolded over the simulation time steps and trained in a way of backpropagation through time (Lee et al., 2016; Shrestha & Orchard, 2018).”
Advancing Neuromorphic Computing With Loihi: A Survey of Results and Outlook (2021, methodology): “Directly trained SNNs benchmarked on Loihi to date have used the Spike Layer Error Reassign-ment (SLAYER) algorithm [22], spatiotemporal backpropagation (STDB) [37], and a combination of BPTT with deep rewiring proposed in [36], which trains recurrent long short-term SNNs (LSNNs) with capabilities…”
Event-Based Vision: A Survey (2019, methodology): “learning for feature extraction, but there is growing interest in finding efficient ways to implement supervised learning directly in SNNs [138], [139] and in embedded devices [140].”
Deep Residual Learning in Spiking Neural Networks (2021, methodology): “The method in the first category computes the gradient by unfolding the network over the simulation timesteps [31, 19, 58, 50, 30, 40], which is similar to the idea of backpropagation through time (BPTT).”
Synaptic Plasticity Dynamics for Deep Continuous Local Learning (DECOLLE) (2018, background): “networks sizes in [21] were limited by GPU memory).”
Training Spiking Neural Networks Using Lessons From Deep Learning (2021, methodology): “This can be generalized to account for multiple spikes [73].”
Learning to Detect Objects with a 1 Megapixel Event Camera (2020, methodology): “Some of these methods, such as Spiking Neural Networks [14, 15, 16, 17], exploit the sparsity of the data and can be applied event by event, to preserve the temporal resolution of the events stream [15, 18, 19, 20].”
The remarkable robustness of surrogate gradient learning for instilling complex function in spiking neural networks (2020, methodology): “…studies that rely on surrogate gradients, the surrogate derivative is normalized to 1 (Bellec et al., 2018; Esser et al., 2016; Neftci et al., 2019; Shrestha & Orchard, 2018; Zenke & Ganguli, 2018) 3a), markedly different from the actual derivative of a spiking threshold which is infinite (see…”
Temporal Efficient Training of Spiking Neural Network via Gradient Re-weighting (2022, methodology): “The output layer’s spike frequency or the average membrane potential increment is commonly used as inference indicators in SNNs (Shrestha & Orchard, 2018; Kim et al., 2019).”
Surrogate Gradient Learning in Spiking Neural Networks: Bringing the Power of Gradient-based optimization to spiking neural networks (2019, methodology): “SLAYER: Spike Layer Error Reassignment [20]; EEDN: energy-efficient deep neuromorphic networks.”
Enabling Deep Spiking Neural Networks with Hybrid Conversion and Spike Timing Dependent Backpropagation (2020, methodology): “Shrestha & Orchard (2018) proposed a gradient function similar to the one proposed in this work.”
Spiking Neural Networks and Their Applications: A Review (2022, methodology): “SLAYER [50] distributes the credit of error back in time in order to solve the drawback of event-based methods.”
Enabling Spike-Based Backpropagation for Training Deep Neural Network Architectures (2019, methodology): “The reference Shrestha and Orchard (2018) employed exponential function for the approximate derivative of neuronal function and developed a credit assignment scheme to calculate the temporal dependencies of error throughout the layers.”
Incorporating Learnable Membrane Time Constant to Enhance Learning of Spiking Neural Networks (2020, methodology): “Recently, the surrogate gradient method was proposed and provided another solution to training multi-layer SNNs [35, 26, 64, 57, 54, 34, 27].”