Surrogate Gradient Learning in Spiking Neural Networks: Bringing the Power of Gradient-based optimization to spiking neural networks

By Emre O. Neftci, Hesham Mostafa, Friedemann Zenke. IEEE Signal Processing Magazine, 2019

1,813 citations recorded by the source corpus.

Spiking neural networks (SNNs) are nature's versatile solution to fault-tolerant, energy-efficient signal processing. To translate these benefits into hardware, a growing number of neuromorphic spiking NN processors have attempted to emulate biological NNs. These developments have created an imminent need for methods and tools that enable such systems to solve real-world signal processing problems. Like conventional NNs, SNNs can be trained on real, domain-specific data; however, their training requires the overcoming of a number of challenges linked to their binary and dynamical nature. This article elucidates step-by-step the problems typically encountered when training SNNs and guides the reader through the key concepts of synaptic plasticity and data-driven learning in the spiking setting. Accordingly, it gives an overview of existing approaches and provides an introduction to surrogate gradient (SG) methods, specifically, as a particularly flexible and efficient method to overcome the aforementioned challenges.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Electrical and Electronic Engineering

View Paper: https://www.semanticscholar.org/paper/409e5a3b7c25cd0cfcf0ff37cc47f7625c09f2fb

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Electrical and Electronic Engineering

Citations in the corpus

DIET-SNN: A Low-Latency Spiking Neural Network With Direct Input Encoding and Leakage and Threshold Optimization (2021, background): “In recent years, the advent of supervised learning algorithms for SNNs has overcome many of the roadblocks surrounding the discontinuous derivative of the spike activation function [12].”
The Heidelberg Spiking Data Sets for the Systematic Evaluation of Spiking Neural Networks (2019, methodology): “In the present manuscript, we trained SNNs using surrogate gradients in combination with BPTT [5]–[8], [62], [63].”
Synaptic Plasticity Dynamics for Deep Continuous Local Learning (DECOLLE) (2018, methodology): “To tackle the temporal dynamics of the neurons, we use a recently established equivalence between SNNs and recurrent ANNs [18].”
Incorporating Learnable Membrane Time Constant to Enhance Learning of Spiking Neural Networks (2020, methodology): “[63, 42] systematically studied the remarkable robustness of surrogate gradient learning and showed that SNNs optimized by the surrogate gradient methods can achieve competitive performance with ANNs.”
Attention Spiking Neural Networks (2022, methodology): “Directly training an SNN is another training mode of SNN, which constitutes a continuous relaxation of the non-smooth spiking to enable backpropagation with a surrogate gradient [36].”
Spike-driven Transformer (2023, background): “Moreover, while keeping its own spike-driven beneﬁts, SNNS have greatly improved its task accuracy by integrating deep learning technologies like network architecture [26, 25], gradient backpropagation [36, 37], normalization [38, 39], etc.”
RMP-SNN: Residual Membrane Potential Neuron for Enabling Deeper High-Accuracy and Low-Latency Spiking Neural Network (2020, methodology): “In order to scale the networks much deeper, spike-based error backpropagation algorithms have been proposed for the supervised training of SNNs [1, 15, 19, 21, 27, 29, 35, 41].”
Spikformer: When Spiking Neural Network Meets Transformer (2022, methodology): “Because the event-triggered mechanism in spiking neurons is non-differentiable, the surrogate gradient is used for backpropagation (Lee et al., 2020; Neftci et al., 2019)Xiao et al. (2021a) adopts implicit differentiation on the equilibrium state to train SNN.”
A review of learning in biologically plausible spiking neural networks (2019, positive): “Recurrent neural networks are a class of neural networks that their internal states evolves with time, and they have been used in temporal processing tasks such as noisy time series prediction, language translation, and automatic speech recognition (Bellec, Salaj, Subramoney, Legenstein, & Maass, 20”
Training Spiking Neural Networks Using Lessons From Deep Learning (2021, background): “While several studies have explored the impact of various surrogates on the learning process [113], [172], our understanding tends to be limited to what is known about biased gradient estimators.”
Enabling Deep Spiking Neural Networks with Hybrid Conversion and Spike Timing Dependent Backpropagation (2020, methodology): “…of the spiking neuron is handled by either approximating the spiking neuron model as continuous and differentiable (Huh & Sejnowski, 2018) or by deﬁning a surrogate gradient as a continuous approximation of the real gradient (Wu et al., 2018; Bellec et al., 2018; Neftci et al., 2019).”
Opportunities for neuromorphic computing algorithms and applications (2022, methodology): “Some of the approaches that adapt deep learning-style training include using a surrogate gradient and having a smoothed activation function to compute the error gradients while performing weight adjustments in each of the successive layers 19,21 .”
Deep Residual Learning in Spiking Neural Networks (2021, methodology): “1 Learning Methods of Spiking Neural Networks ANN to SNN conversion (ANN2SNN) [20, 4, 46, 49, 12, 11, 6, 54, 33] and backpropagation with surrogate gradient [40] are the two main methods to get deep SNNs.”
The remarkable robustness of surrogate gradient learning for instilling complex function in spiking neural networks (2020, background): “The scale of the substituted derivative has a strong effect on the surrogate gradient due to both explicit and implicit forms of recurrence in SNNs (see Figure 4; Neftci et al., 2019).”
Temporal Efficient Training of Spiking Neural Network via Gradient Re-weighting (2022, background): “In this area, SNNs are regarded as special RNNs and training with BPTT (Neftci et al., 2019).”