Spiking Deep Networks with LIF Neurons

By Eric Hunsberger, Chris Eliasmith. arXiv.org, 2015

297 citations recorded by the source corpus.

We train spiking deep networks using leaky integrate-and-fire (LIF) neurons, and achieve state-of-the-art results for spiking networks on the CIFAR-10 and MNIST datasets. This demonstrates that biologically-plausible spiking LIF neurons can be integrated into deep networks can perform as well as other spiking models (e.g. integrate-and-fire). We achieved this result by softening the LIF response function, such that its derivative remains bounded, and by training the network with noise to provide robustness against the variability introduced by spikes. Our method is general and could be applied to other neuron types, including those used on modern neuromorphic hardware. Our work brings more biological realism into modern image classification models, with the hope that these models can inform how the brain performs this difficult task. It also provides new methods for training deep networks to run on neuromorphic hardware, with the aim of fast, power-efficient image classification for robotics applications.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Electrical and Electronic Engineering

View Paper: https://www.semanticscholar.org/paper/0a76cdb10b244d90770888f72ee46ca3ea1d918d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Electrical and Electronic Engineering

Citations in the corpus

Towards spike-based machine intelligence with neuromorphic computing (2019, background): “Recent works 29–31 have proposed practical solutions to determine optimal firing rates, and additional constraints (such as noise or leaky ReLUs) are introduced during training of the DLN to better match the spiking neuron’s firing rate 36 .”
Training Spiking Neural Networks Using Lessons From Deep Learning (2021, methodology): “The high-precision activation function of each neuron is converted into either a spike rate [155], [156], [157], [158], [159] or a latency code [160].”
Surrogate Gradient Learning in Spiking Neural Networks: Bringing the Power of Gradient-based optimization to spiking neural networks (2019, methodology): “For instance, Hunsberger and Eliasmith [30] as well as Neftci et al.”
Spikformer: When Spiking Neural Network Meets Transformer (2022, methodology): “In ANN-to-SNN conversion (Cao et al., 2015; Hunsberger & Eliasmith, 2015; Rueckauer et al., 2017; Bu et al., 2021; Meng et al., 2022; Wang et al., 2022), the high-performance pre-trained ANN is converted to SNN by replacing the ReLU activation layers with spiking neurons.”
RMP-SNN: Residual Membrane Potential Neuron for Enabling Deeper High-Accuracy and Low-Latency Spiking Neural Network (2020, methodology): “A notable exception to ReLU-IF mapping is the work of Hunsberger et al. [13] who used more bio-plausible Leaky-Integrate-and-Fire (LIF) neuron during inference by training the ANN with rate-based soft-LIF non-linearity.”
The remarkable robustness of surrogate gradient learning for instilling complex function in spiking neural networks (2020, methodology): “One approach is to use analog-valued machine learning data sets as input currents directly (Hunsberger & Eliasmith, 2015) or to first convert them to Poisson input spike trains (Lee et al., 2016; Pfeiffer & Pfeil, 2018).”
The Heidelberg Spiking Data Sets for the Systematic Evaluation of Spiking Neural Networks (2019, background): “However, it is essential to realize that there exists a plethora of alternative gradient-based approaches based on network translation [3], [64]–[66], single spike timing [23], [67], mean firing rate [68], [69], and stochastic approximations [37], [70]–[72].”
Direct Training for Spiking Neural Networks: Faster, Larger, Better (2018, background): “Many normalization techniques have been proposed to improve the convergence (Ioffe and Szegedy 2015; Wu and He 2018).”
SLAYER: Spike Layer Error Reassignment in Time (2018, background): “…accuracy such as introducing extra constraints on neuron ﬁring rate [23], scaling the weights [23–25], constraining the network parameters [20], formulating an equivalent transfer function for a spiking neuron [19–22], adding noise in the model [21,22], using probabilistic weights [18] and so on.”
Spatio-Temporal Backpropagation for Training High-Performance Spiking Neural Networks (2017, background): “The second one ﬁrstly trains an ANN, and then transforms it into its SNN version with the same network structure where the spiking rate of SNN neurons acts as the analog activity of ANN neurons [18]–[21].”
STDP-based spiking deep neural networks for object recognition (2016, background): “Recent supervised DCNNs have reached high accuracies on the most challenging object recognition datasets such as Imagenet.”
Enabling Deep Spiking Neural Networks with Hybrid Conversion and Spike Timing Dependent Backpropagation (2020, methodology): “Consequently, there has been several works over the past few years that propose different learning algorithms or learning rules for implementing deep convolutional SNNs for complex visual recognition tasks (Wu et al., 2019; Hunsberger & Eliasmith, 2015; Cao et al., 2015).”
Training Deep Spiking Neural Networks Using Backpropagation (2016, methodology): “An alternative approach has recently been introduced by [23], in which a SNN learns from spikes, but requires keeping statistics for computing stochastic gradient descent (SGD) updates in order to approximate a conventional ANN.”
Deep Learning in Spiking Neural Networks (2018, background): “Many studies have shown high performance of converted spiking CNNs (close to conventional CNNs) while using fewer operations and consuming less energy [177], [178], [141], [179], which enable the deep CNNs to be implemented on hardware [180], [181], [182].”
Deep Residual Learning in Spiking Neural Networks (2021, methodology): “The ANN2SNN method ﬁrstly trains an ANN with ReLU activation, then converts the ANN to an SNN by replacing ReLU with spiking neurons and adding scaling operations like weight normalization and threshold balancing.”
Going Deeper With Directly-Trained Larger Spiking Neural Networks (2020, methodology): “2019) improves the leaky integrate-and-fire (LIF) model (Hunsberger and Eliasmith 2015) to an iterative LIF model and develop STBP learning algorithm, which makes it friendly on ANNoriented programming frameworks and speeds up the training process.”