Direct Training for Spiking Neural Networks: Faster, Larger, Better

By Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, Luping P. Shi. AAAI Conference on Artificial Intelligence, 2018

797 citations recorded by the source corpus.

Spiking neural networks (SNNs) that enables energy efficient implementation on emerging neuromorphic hardware are gaining more attention. Yet now, SNNs have not shown competitive performance compared with artificial neural networks (ANNs), due to the lack of effective learning algorithms and efficient programming frameworks. We address this issue from two aspects: (1) We propose a neuron normalization technique to adjust the neural selectivity and develop a direct learning algorithm for deep SNNs. (2) Via narrowing the rate coding window and converting the leaky integrate-and-fire (LIF) model into an explicitly iterative version, we present a Pytorch-based implementation method towards the training of large-scale SNNs. In this way, we are able to train deep SNNs with tens of times speedup. As a result, we achieve significantly better accuracy than the reported works on neuromorphic datasets (N-MNIST and DVSCIFAR10), and comparable accuracy as existing ANNs and pre-trained SNNs on non-spiking datasets (CIFAR10). To our best knowledge, this is the first work that demonstrates direct training of deep SNNs with high performance on CIFAR10, and the efficient implementation provides a new way to explore the potential of SNNs.

Subfields: Cognitive Neuroscience, Electrical and Electronic Engineering

View Paper: https://www.semanticscholar.org/paper/67b12aa7cde36523780c6be84afb9b95ddbd078a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Electrical and Electronic Engineering

Citations in the corpus

Enabling Deep Spiking Neural Networks with Hybrid Conversion and Spike Timing Dependent Backpropagation (2020, background): “Consequently, there has been several works over the past few years that propose different learning algorithms or learning rules for implementing deep convolutional SNNs for complex visual recognition tasks (Wu et al., 2019; Hunsberger & Eliasmith, 2015; Cao et al., 2015).”
Spiking-YOLO: Spiking Neural Network for Energy-Efficient Object Detection (2019, background): “However, they are difﬁcult to train which has been one of the major obstacles when deploying SNNs in various applications (Wu et al. 2019).”
Enabling Spike-Based Backpropagation for Training Deep Neural Network Architectures (2019, methodology): “In addition, as opposed to complex error or neuron normalization method adopted by [24] and [47], respectively, we demonstrate that deep SNNs can be naturally trained by only accounting for spiking activities of the network.”
Temporal Efficient Training of Spiking Neural Network via Gradient Re-weighting (2022, methodology): “We adopt the Leaky Integrate-and-Fire (LIF) model and translate it to an iterative expression with the Euler method (Wu et al., 2019).”
Incorporating Learnable Membrane Time Constant to Enhance Learning of Spiking Neural Networks (2020, methodology): “Similar to [50, 58, 47], the input is directly fed to our network without being firstly converted to spikes.”
Spike-driven Transformer (2023, methodology): “We employ the direct training method, using the ﬁrst SNN layer as the spike encoding layer and applying surrogate gradient training [42].”