Going Deeper With Directly-Trained Larger Spiking Neural Networks

By Hanle Zheng, Yujie Wu, Lei Deng, Yifan Hu, Guoqi Li. AAAI Conference on Artificial Intelligence, 2020

736 citations recorded by the source corpus.

Spiking neural networks (SNNs) are promising in a bio-plausible coding for spatio-temporal information and event-driven signal processing, which is very suited for energy-efficient implementation in neuromorphic hardware. However, the unique working mode of SNNs makes them more difficult to train than traditional networks. Currently, there are two main routes to explore the training of deep SNNs with high performance. The first is to convert a pre-trained ANN model to its SNN version, which usually requires a long coding window for convergence and cannot exploit the spatio-temporal features during training for solving temporal tasks. The other is to directly train SNNs in the spatio-temporal domain. But due to the binary spike activity of the firing function and the problem of gradient vanishing or explosion, current methods are restricted to shallow architectures and thereby difficult in harnessing large-scale datasets (e.g. ImageNet). To this end, we propose a threshold-dependent batch normalization (tdBN) method based on the emerging spatio-temporal backpropagation, termed “STBP-tdBN”, enabling direct training of a very deep SNN and the efficient implementation of its inference on neuromorphic hardware. With the proposed method and elaborated shortcut connection, we significantly extend directly-trained SNNs from a shallow structure (

Subfields: Cognitive Neuroscience, Electrical and Electronic Engineering

View Paper: https://www.semanticscholar.org/paper/e5ac393af264eb99484973c565ce2e1f12c85e18

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

1. Yonghong Tian
2. Guoqi Li
3. Liuliang Yuan
4. Man Yao
5. Timothée Masquelier
6. Zhaofei Yu
7. Wei Fang
8. Yanqing Chen

In Electrical and Electronic Engineering

1. Yonghong Tian
2. Guoqi Li
3. Liuliang Yuan
4. Man Yao
5. Timothée Masquelier
6. Zhaofei Yu
7. Wei Fang
8. Yanqing Chen

Citations in the corpus

• Spikformer: When Spiking Neural Network Meets Transformer (2022, methodology): “With the development of artificial neural networks (ANNs), SNNs are able to lift performance by borrowing advanced architectures from ANNs, such as ResNet-like SNNs (Hu et al., 2021a; Fang et al., 2021a; Zheng et al., 2021; Hu et al., 2021b), Spiking Recurrent Neural Networks (Lotfi Rezaabad & Vishw”
• Deep Residual Learning in Spiking Neural Networks (2021, methodology): “The basic block of Spiking ResNet used in [64, 17, 30] simply mimics the block in ANNs by replacing ReLU activation layers with spiking neurons (SN), which is illustrated in Fig.”
• Hebbian Learning based Orthogonal Projection for Continual Learning of Spiking Neural Networks (2024, methodology): “A.2 BPTT WITH SG Backpropagation through time (BPTT) with surrogate gradients (SG) is a popular way to train SNNs without specific coding schemes (Shrestha & Orchard, 2018; Bellec et al., 2018; Wu et al., 2018; Zheng et al., 2021; Li et al., 2021; Fang et al., 2021; Yin et al., 2021).”
• Temporal Efficient Training of Spiking Neural Network via Gradient Re-weighting (2022, methodology): “The network architectures in this paper include ResNet-19 (Zheng et al., 2021), Spiking-ResNet34 (Zheng et al., 2021), SEW-ResNet34 (Fang et al., 2021), SNN-5, and VGGSNN. SNN-5 (16C3-64C5-AP2-128C5-AP2-256C5-AP2-512C3-AP2-FC) is a simple convolutional SNN suitable for multiple runs to discover stat”
• Attention Spiking Neural Networks (2022, methodology): “Vanilla Res-SNN [15], executing the same shortcut and residual block as Res-CNN.”
• Spike-driven Transformer (2023, background): “Vanilla Res-SNN [38], similar to vanilla Res-CNN [67], performs a shortcut between membrane potential and spike.”