Spikformer: When Spiking Neural Network Meets Transformer

By Zhaokun Zhou, Yuesheng Zhu, Chao He, Yaowei Wang, Shuicheng Yan, Yonghong Tian, Liuliang Yuan. International Conference on Learning Representations, 2022

519 citations recorded by the source corpus.

We consider two biologically plausible structures, the Spiking Neural Network (SNN) and the self-attention mechanism. The former offers an energy-efficient and event-driven paradigm for deep learning, while the latter has the ability to capture feature dependencies, enabling Transformer to achieve good performance. It is intuitively promising to explore the marriage between them. In this paper, we consider leveraging both self-attention capability and biological properties of SNNs, and propose a novel Spiking Self Attention (SSA) as well as a powerful framework, named Spiking Transformer (Spikformer). The SSA mechanism in Spikformer models the sparse visual feature by using spike-form Query, Key, and Value without softmax. Since its computation is sparse and avoids multiplication, SSA is efficient and has low computational energy consumption. It is shown that Spikformer with SSA can outperform the state-of-the-art SNNs-like frameworks in image classification on both neuromorphic and static datasets. Spikformer (66.3M parameters) with comparable size to SEW-ResNet-152 (60.2M,69.26%) can achieve 74.81% top1 accuracy on ImageNet using 4 time steps, which is the state-of-the-art in directly trained SNNs models.

Subfields: Cognitive Neuroscience, Electrical and Electronic Engineering

View Paper: https://www.semanticscholar.org/paper/ffd91f85d6d19e75309ececfd190afa6bb562284

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

1. Yonghong Tian
2. Liuliang Yuan
3. Man Yao
4. Guoqi Li
5. Boxing Xu
6. Zhaokun Zhou
7. Jiakui Hu

In Electrical and Electronic Engineering

1. Yonghong Tian
2. Liuliang Yuan
3. Man Yao
4. Boxing Xu
5. Zhaokun Zhou
6. Jiakui Hu
7. Guoqi Li

Citations in the corpus

• Spike-driven Transformer (2023, methodology): “The typical spiking self-attentions in the current spiking Transformers [20, 19] would convert Q , K , V into spike-form before performing two matrix multiplications similar to those in VSA.”