Real-time classification and sensor fusion with a spiking deep belief network

By Peter O'Connor, Daniel Neil, Shih-Chii Liu, Tobi Delbruck, Michael Pfeiffer. 2013

390 citations recorded by the source corpus.

Deep Belief Networks (DBNs) have recently shown impressive performance on a broad range of classification problems. Their generative properties allow better understanding of the performance, and provide a simpler solution for sensor fusion tasks. However, because of their inherent need for feedback and parallel update of large numbers of units, DBNs are expensive to implement on serial computers. This paper proposes a method based on the Siegert approximation for Integrate-and-Fire neurons to map an offline-trained DBN onto an efficient event-driven spiking neural network suitable for hardware implementation. The method is demonstrated in simulation and by a real-time implementation of a 3-layer network with 2694 neurons used for visual classification of MNIST handwritten digits with input from a 128 × 128 Dynamic Vision Sensor (DVS) silicon retina, and sensory-fusion using additional input from a 64-channel AER-EAR silicon cochlea. The system is implemented through the open-source software in the jAER project and runs in real-time on a laptop computer. It is demonstrated that the system can recognize digits in the presence of distractions, noise, scaling, translation and rotation, and that the degradation of recognition performance by using an event-based approach is less than 1%. Recognition is achieved in an average of 5.8 ms after the onset of the presentation of a digit. By cue integration from both silicon retina and cochlea outputs we show that the system can be biased to select the correct digit from otherwise ambiguous input.

Subfields: Cognitive Neuroscience, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Electrical and Electronic Engineering

Citations in the corpus

Phased LSTM: Accelerating Recurrent Network Training for Long or Event-based Sequences (2016, methodology): “In previous work using event-based input data [21, 23], the timing information was sometimes removed and instead a frame-based representation was generated by computing the pixel-wise event-rate over some time period (as shown in Fig.”
Event-Based Vision: A Survey (2019, methodology): “reconstruct the input, as was done in [134] with a Deep Belief Network (DBN).”
Spatio-Temporal Backpropagation for Training High-Performance Spiking Neural Networks (2017, background): “Spiking RBM(pre-training*) (Peter et al., 2013) 784-500-500-10 None 97.”
SLAYER: Spike Layer Error Reassignment in Time (2018, methodology): “There are different approaches to overcome the loss of accuracy such as introducing extra constraints on neuron firing rate [23], scaling the weights [23–25], constraining the network parameters [20], formulating an equivalent transfer function for a spiking neuron [19–22], adding noise in the model”
Effective sensor fusion with event-based sensors and deep network architectures (2016, methodology): “The benefits of these networks including robustness to noise, bit precision of the hardware platform, and efficient processing, have been demonstrated in some studies [2], [3].”
Supervised Learning Based on Temporal Coding in Spiking Neural Networks (2016, methodology): “Spiking networks are dynamical systems in which time plays a crucial role, while time is abstracted away in conventional feedforward ANNs.”
Spiking Deep Networks with LIF Neurons (2015, background): “There has recently been considerable effort to take deep ANNs and make them more biologically plausible by introducing neural “spiking” [8, 9, 10, 11, 12, 13], such that connected nodes in the network transmit information via instantaneous single bits (spikes), rather than transmitting realvalued ac”
Retinomorphic Event-Based Vision Sensors: Bioinspired Cameras With Spiking Output (2014, background): “[69] have shown handwritten character recognition using event-driven stochastic encoding learning and processing.”
Unsupervised learning of digit recognition using spike-timing-dependent plasticity (2015, methodology): “Typically, the training procedure used for such rate-based training is based on popular models in machine learning like the Restricted Boltzman Machine (RBM) or convolutional neural networks.”
Minitaur, an Event-Driven FPGA-Based Spiking Network Accelerator (2014, methodology): “2% accuracy on the MNIST task with LIF neurons in software [7] and shown in Fig.”
Memory and Information Processing in Neuromorphic Systems (2015, methodology): “However, it has been argued that there are several areas in which neuromorphic systems offer significant advantages over conventional computers [13], [14], [27], [4], [5], such as that of sensory processing [28] or “autonomous systems”.”