Rethinking the performance comparison between SNNS and ANNS

By Lei Deng, Yujie Wu, Xing Hu, Ling Liang, Yufei Ding, Guoqi Li, Guangshe Zhao, Peng Li, Yuan Xie. Neural Networks, 2020

355 citations recorded by the source corpus.

Artificial neural networks (ANNs), a popular path towards artificial intelligence, have experienced remarkable success via mature models, various benchmarks, open-source datasets, and powerful computing platforms. Spiking neural networks (SNNs), a category of promising models to mimic the neuronal dynamics of the brain, have gained much attention for brain inspired computing and been widely deployed on neuromorphic devices. However, for a long time, there are ongoing debates and skepticisms about the value of SNNs in practical applications. Except for the low power attribute benefit from the spike-driven processing, SNNs usually perform worse than ANNs especially in terms of the application accuracy. Recently, researchers attempt to address this issue by borrowing learning methodologies from ANNs, such as backpropagation, to train high-accuracy SNN models. The rapid progress in this domain continuously produces amazing results with ever-increasing network size, whose growing path seems similar to the development of deep learning. Although these ways endow SNNs the capability to approach the accuracy of ANNs, the natural superiorities of SNNs and the way to outperform ANNs are potentially lost due to the use of ANN-oriented workloads and simplistic evaluation metrics. In this paper, we take the visual recognition task as a case study to answer the questions of "what workloads are ideal for SNNs and how to evaluate SNNs makes sense". We design a series of contrast tests using different types of datasets (ANN-oriented and SNN-oriented), diverse processing models, signal conversion methods, and learning algorithms. We propose comprehensive metrics on the application accuracy and the cost of memory & compute to evaluate these models, and conduct extensive experiments. We evidence the fact that on ANN-oriented workloads, SNNs fail to beat their ANN counterparts; while on SNN-oriented workloads, SNNs can fully perform better. We further demonstrate that in SNNs there exists a trade-off between the application accuracy and the execution cost, which will be affected by the simulation time window and firing threshold. Based on these abundant analyses, we recommend the most suitable model for each scenario. To the best of our knowledge, this is the first work using systematical comparisons to explicitly reveal that the straightforward workload porting from ANNs to SNNs is unwise although many works are doing so and a comprehensive evaluation indeed matters. Finally, we highlight the urgent need to build a benchmarking framework for SNNs with broader tasks, datasets, and metrics.

Subfields: Artificial Intelligence, Control and Systems Engineering, Electrical and Electronic Engineering

View Paper: https://www.semanticscholar.org/paper/7b880fe48a1c34c27697a798e9b6567551f97d68

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Lei Deng
2. Yuhang Li
3. Guoqi Li
4. Zhaofei Yu
5. Shi Gu
6. Wei Fang
7. Yanqing Chen
8. Shanghang Zhang

In Electrical and Electronic Engineering

1. Yonghong Tian
2. Guoqi Li
3. Man Yao
4. Timothée Masquelier
5. Zhaofei Yu
6. Wei Fang
7. Yanqing Chen
8. Tiejun Huang

Citations in the corpus

• Attention Spiking Neural Networks (2022, methodology): “Static Images: For the analog-valued signal of pixel intensity in images, adding an encoding layer to generate spike signals globally is a generic method of SNN [45].”
• Temporal Efficient Training of Spiking Neural Network via Gradient Re-weighting (2022, methodology): “Nevertheless, it is commonly agreed that the direct training method for high-performance SNN is still crucial since it distinguishes SNNs from converted ANNs, especially on neuromorphic datasets.”
• Incorporating Learnable Membrane Time Constant to Enhance Learning of Spiking Neural Networks (2020, methodology): “To evaluate the encoder, we give inputs xt to the trained network and show the output spikes St (c) and the firing rates F nTs(c) = 1 Ts ∑Ts−1 t=0 S n t (c) from channel c in the n-th layer, which is similar to [10].”