Accurate and efficient time-domain classification with adaptive spiking recurrent neural networks

By Bojian Yin, Federico Corradi, Sander M. Bohte. Nature Machine Intelligence, 2021

321 citations recorded by the source corpus.

Inspired by detailed modelling of biological neurons, spiking neural networks (SNNs) are investigated as biologically plausible and high-performance models of neural computation. The sparse and binary communication between spiking neurons potentially enables powerful and energy-efficient neural networks. The performance of SNNs, however, has remained lacking compared with artificial neural networks. Here we demonstrate how an activity-regularizing surrogate gradient combined with recurrent networks of tunable and adaptive spiking neurons yields the state of the art for SNNs on challenging benchmarks in the time domain, such as speech and gesture recognition. This also exceeds the performance of standard classical recurrent neural networks and approaches that of the best modern artificial neural networks. As these SNNs exhibit sparse spiking, we show that they are theoretically one to three orders of magnitude more computationally efficient compared to recurrent neural networks with similar performance. Together, this positions SNNs as an attractive solution for AI hardware implementations. The use of sparse signals in spiking neural networks, modelled on biological neurons, offers in principle a highly efficient approach for artificial neural networks when implemented on neuromorphic hardware, but new training approaches are needed to improve performance. Using a new type of activity-regularizing surrogate gradient for backpropagation combined with recurrent networks of tunable and adaptive spiking neurons, state-of-the-art performance for spiking neural networks is demonstrated on benchmarks in the time domain.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Electrical and Electronic Engineering

View Paper: https://www.semanticscholar.org/paper/d92073e72bcf096a52fd2400459f64d1800a063f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Lei Deng
2. Guoqi Li
3. Qingyan Meng
4. Mingqing Xiao
5. Zongpeng Zhang
6. Liuliang Yuan
7. D.K. He
8. Zhouchen Lin

In Cognitive Neuroscience

1. Yonghong Tian
2. Liuliang Yuan
3. Man Yao
4. Guoqi Li
5. Lei Deng
6. Boxing Xu
7. Zhaokun Zhou
8. Jiakui Hu

In Electrical and Electronic Engineering

1. Yonghong Tian
2. Liuliang Yuan
3. Man Yao
4. Guoqi Li
5. Lei Deng
6. Boxing Xu
7. Zhaokun Zhou
8. Jiakui Hu

Citations in the corpus

• Attention Spiking Neural Networks (2022, methodology): “works [12], [50], we assume the data for various operations are 32-bit floating-point implementation in 45 nm technology [51], in which EMAC = 4.”
• Spike-driven Transformer (2023, result): “This is consistent with other neuroscience-inspired methods to optimize the internal dynamics of SNNs, such as complex spiking neuron design [69], attention mechanism [27], long short-term memory [34], recurrent connection [70], information maximization [71], etc. • Dynamical isometry .”
• Hebbian Learning based Orthogonal Projection for Continual Learning of Spiking Neural Networks (2024, methodology): “A.2 BPTT WITH SG Backpropagation through time (BPTT) with surrogate gradients (SG) is a popular way to train SNNs without specific coding schemes (Shrestha & Orchard, 2018; Bellec et al., 2018; Wu et al., 2018; Zheng et al., 2021; Li et al., 2021; Fang et al., 2021; Yin et al., 2021).”