Deep Residual Learning in Spiking Neural Networks

By Wei Fang, Zhaofei Yu, Yanqing Chen, Tiejun Huang, Timothée Masquelier, Yonghong Tian. Neural Information Processing Systems, 2021

793 citations recorded by the source corpus.

Deep Spiking Neural Networks (SNNs) present optimization difficulties for gradient-based approaches due to discrete binary activation and complex spatial-temporal dynamics. Considering the huge success of ResNet in deep learning, it would be natural to train deep SNNs with residual learning. Previous Spiking ResNet mimics the standard residual block in ANNs and simply replaces ReLU activation layers with spiking neurons, which suffers the degradation problem and can hardly implement residual learning. In this paper, we propose the spike-element-wise (SEW) ResNet to realize residual learning in deep SNNs. We prove that the SEW ResNet can easily implement identity mapping and overcome the vanishing/exploding gradient problems of Spiking ResNet. We evaluate our SEW ResNet on ImageNet, DVS Gesture, and CIFAR10-DVS datasets, and show that SEW ResNet outperforms the state-of-the-art directly trained SNNs in both accuracy and time-steps. Moreover, SEW ResNet can achieve higher performance by simply adding more layers, providing a simple method to train deep SNNs. To our best knowledge, this is the first time that directly training deep SNNs with more than 100 layers becomes possible. Our codes are available at https://github.com/fangwei123456/Spike-Element-Wise-ResNet.

Subfields: Cognitive Neuroscience, Electrical and Electronic Engineering

View Paper: https://www.semanticscholar.org/paper/35fcc65db264d9c73ed19d0f3f9d53a43991bdbf

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

1. Yonghong Tian
2. Liuliang Yuan
3. Man Yao
4. Guoqi Li
5. Yuhang Li
6. Shi-Wee Deng
7. Shi Gu
8. Shanghang Zhang

In Electrical and Electronic Engineering

1. Yonghong Tian
2. Liuliang Yuan
3. Man Yao
4. Yuhang Li
5. Shi-Wee Deng
6. Shi Gu
7. Shanghang Zhang
8. Guoqi Li

Citations in the corpus

• Temporal Efficient Training of Spiking Neural Network via Gradient Re-weighting (2022, methodology): “The network architectures in this paper include ResNet-19 (Zheng et al., 2021), Spiking-ResNet34 (Zheng et al., 2021), SEW-ResNet34 (Fang et al., 2021), SNN-5, and VGGSNN. SNN-5 (16C3-64C5-AP2-128C5-AP2-256C5-AP2-512C3-AP2-FC) is a simple convolutional SNN suitable for multiple runs to discover stat”
• Attention Spiking Neural Networks (2022, methodology): “Recently, by redesigning the connection method of the shortcut, two works [12], [14] have successfully trained more than 100 layers of Res-SNN on ImageNet-1 K.”
• Spikformer: When Spiking Neural Network Meets Transformer (2022, methodology): “…neural networks (ANNs), SNNs are able to lift performance by borrowing advanced architectures from ANNs, such as ResNet-like SNNs (Hu et al., 2021a; Fang et al., 2021a; Zheng et al., 2021; Hu et al., 2021b), Spiking Recurrent Neural Networks (Lotfi Rezaabad & Vishwanath, 2020) and Spiking Graph…”
• Spike-driven Transformer (2023, methodology): “Moreover, while keeping its own spike-driven benefits, SNNS have greatly improved its task accuracy by integrating deep learning technologies like network architecture [26, 25], gradient backpropagation [36, 37], normalization [38, 39], etc.”
• Hebbian Learning based Orthogonal Projection for Continual Learning of Spiking Neural Networks (2024, methodology): “A.2 BPTT WITH SG Backpropagation through time (BPTT) with surrogate gradients (SG) is a popular way to train SNNs without specific coding schemes (Shrestha & Orchard, 2018; Bellec et al., 2018; Wu et al., 2018; Zheng et al., 2021; Li et al., 2021; Fang et al., 2021; Yin et al., 2021).”
• Training Spiking Neural Networks Using Lessons From Deep Learning (2021, background): “You might see this referred to as the “arctan” surrogate gradient, first proposed in [171].”