Attention Spiking Neural Networks
By Man Yao, Guangshe Zhao, Hengyu Zhang, Yifan Hu, Lei Deng, Yonghong Tian, Bo Xu, Guoqi Li. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022
303 citations recorded by the source corpus.
Brain-inspired spiking neural networks (SNNs) are becoming a promising energy-efficient alternative to traditional artificial neural networks (ANNs). However, the performance gap between SNNs and ANNs has been a significant hindrance to deploying SNNs ubiquitously. To leverage the full potential of SNNs, in this paper we study the attention mechanisms, which can help human focus on important information. We present our idea of attention in SNNs with a multi-dimensional attention module, which infers attention weights along the temporal, channel, as well as spatial dimension separately or simultaneously. Based on the existing neuroscience theories, we exploit the attention weights to optimize membrane potentials, which in turn regulate the spiking response. Extensive experimental results on event-based action recognition and image classification datasets demonstrate that attention facilitates vanilla SNNs to achieve sparser spiking firing, better performance, and energy efficiency concurrently. In particular, we achieve top-1 accuracy of 75.92% and 77.08% on ImageNet-1 K with single/4-step Res-SNN-104, which are state-of-the-art results in SNNs. Compared with counterpart Res-ANN-104, the performance gap becomes -0.95/+0.21 percent and the energy efficiency is 31.8×/7.4×. To analyze the effectiveness of attention SNNs, we theoretically prove that the spiking degradation or the gradient vanishing, which usually holds in general SNNs, can be resolved by introducing the block dynamical isometry theory. We also analyze the efficiency of attention SNNs based on our proposed spiking response visualization method. Our work lights up SNN's potential as a general backbone to support various applications in the field of SNN research, with a great balance between effectiveness and energy efficiency.
Subfields: Cellular and Molecular Neuroscience, Cognitive Neuroscience, Electrical and Electronic Engineering
View Paper: https://www.semanticscholar.org/paper/9c5cd94bbe92b831b12149b9504c25afc8d00d3d
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Cellular and Molecular Neuroscience
In Cognitive Neuroscience
In Electrical and Electronic Engineering
Citations in the corpus
- SpikingJelly: An open-source machine learning infrastructure platform for spike-based intelligence (2023, background): “…used in many spiking deep learning studies, including adversarial attack ( 100 , 101 ), ANN2SNN ( 95 , 102 – 106 ), attention mechanisms ( 107 , 108 ), depth estimation from DVS data ( 69 , 109 ), development of innovative materials ( 110 ), emotion recognition ( 111 ), energy estimation ( 112…”
- Spike-driven Transformer (2023, methodology): “Refer to previous works[91, 70, 92, 27], we assume the data for various operations are 32-bit floating point implementation in 45nm technology [93], in which E MAC = 4 .”