Supervised Learning Based on Temporal Coding in Spiking Neural Networks

By Hesham Mostafa. IEEE Transactions on Neural Networks and Learning Systems, 2016

398 citations recorded by the source corpus.

Gradient descent training techniques are remarkably successful in training analog-valued artificial neural networks (ANNs). Such training techniques, however, do not transfer easily to spiking networks due to the spike generation hard nonlinearity and the discrete nature of spike communication. We show that in a feedforward spiking network that uses a temporal coding scheme where information is encoded in spike times instead of spike rates, the network input–output relation is differentiable almost everywhere. Moreover, this relation is piecewise linear after a transformation of variables. Methods for training ANNs thus carry directly to the training of such spiking networks as we show when training on the permutation invariant MNIST task. In contrast to rate-based spiking networks that are often used to approximate the behavior of ANNs, the networks we present spike much more sparsely and their behavior cannot be directly approximated by conventional ANNs. Our results highlight a new approach for controlling the behavior of spiking networks with realistic temporal dynamics, opening up the potential for using these networks to process spike patterns with complex temporal information.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Electrical and Electronic Engineering

View Paper: https://www.semanticscholar.org/paper/83e51128c4315799854ee645dac576d9c8331756

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Electrical and Electronic Engineering

Citations in the corpus

Enabling Spike-Based Backpropagation for Training Deep Neural Network Architectures (2019, background): “, 2018a,b) and “spike-based backpropagation” based supervised learning (Lee et al., 2016; Neftci et al., 2017; Jin et al., 2018;Mostafa, 2018;Wu et al., 2018b).”
The remarkable robustness of surrogate gradient learning for instilling complex function in spiking neural networks (2020, methodology): “previous SNN studies using temporal coding (Mostafa, 2018).”
Deep Learning in Spiking Neural Networks (2018, methodology): “Using the same neural architecture of SpikeProp, new formulations of temporal spike coding and spike time errors have recently improved the spiking backpropagation algorithm (Liu et al., 2017; Mostafa, 2017).”
DIET-SNN: A Low-Latency Spiking Neural Network With Direct Input Encoding and Leakage and Threshold Optimization (2021, methodology): “Some proposals adopt kernel-based spike response models [25]–[27], but for the most part, these approaches show limited performance on simple datasets and do not scale for deep networks.”
Hebbian Learning based Orthogonal Projection for Continual Learning of Spiking Neural Networks (2024, background): “…feedforward inputs and responses for lateral circuits. et al., 2016; Zhang & Li, 2019; Zhou et al., 2021; Wu et al., 2021; Xiao et al., 2021; 2023) or consider the first spiking time as the encoding and derive the analytical transformations (Mostafa, 2017; Comsa et al., 2020; Zhou et al., 2021).”
Surrogate Gradient Learning in Spiking Neural Networks: Bringing the Power of Gradient-based optimization to spiking neural networks (2019, methodology): “Figure 7(b) shows the network’s response after training (see [34] for details on the training process).”
The Heidelberg Spiking Data Sets for the Systematic Evaluation of Spiking Neural Networks (2019, background): “However, it is essential to realize that there exists a plethora of alternative gradient-based approaches based on network translation [3], [64]–[66], single spike timing [23], [67], mean firing rate [68], [69], and stochastic approximations [37], [70]–[72].”
A review of learning in biologically plausible spiking neural networks (2019, neutral): “The rate coding can mask the temporal information that can be processed by DSNN (Lee et al., 2019; Mostafa, 2018).”
Deep Learning With Spiking Neurons: Opportunities and Challenges (2018, methodology): “The sensors record temporally precise information from the environment and SNNs can utilize eﬃcient temporal codes in their computations as well (Mostafa, 2018).”
Deep Residual Learning in Spiking Neural Networks (2021, methodology): “The second method computes the gradients of the timings of existing spikes with respect to the membrane potential at the spike timing [5, 39, 24, 65, 63].”
Deep Supervised Learning Using Local Errors (2017, methodology): “Several recent approaches have applied the backpropagation algorithm to learning precise spike times in spiking neural networks (Mostafa, 2017; Zenke and Ganguli, 2017).”