Converting Static Image Datasets to Spiking Neuromorphic Datasets Using Saccades

By Garrick Orchard, Ajinkya Jayawant, Gregory Cohen, Nitish V. Thakor. 2015

1,000 citations recorded by the source corpus.

Creating datasets for Neuromorphic Vision is a challenging task. A lack of available recordings from Neuromorphic Vision sensors means that data must typically be recorded specifically for dataset creation rather than collecting and labeling existing data. The task is further complicated by a desire to simultaneously provide traditional frame-based recordings to allow for direct comparison with traditional Computer Vision algorithms. Here we propose a method for converting existing Computer Vision static image datasets into Neuromorphic Vision datasets using an actuated pan-tilt camera platform. Moving the sensor rather than the scene or image is a more biologically realistic approach to sensing and eliminates timing artifacts introduced by monitor updates when simulating motion on a computer monitor. We present conversion of two popular image datasets (MNIST and Caltech101) which have played important roles in the development of Computer Vision, and we provide performance metrics on these datasets using spike-based recognition algorithms. This work contributes datasets for future use in the field, as well as results from spike-based algorithms against which future works can compare. Furthermore, by converting datasets already popular in Computer Vision, we enable more direct comparison with frame-based approaches.

Subfields: Cognitive Neuroscience, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Electrical and Electronic Engineering

Citations in the corpus

EMNIST: an extension of MNIST to handwritten letters (2017, methodology): “ and used dataset in the computer vision and neural networks community. However, a good dataset needs to represent a sufﬁciently challenging problem to make it both useful and to ensure its longevity [5]. This is perhaps where MNIST has suffered in the face of the increasingly high accuracies achiev”
Phased LSTM: Accelerating Recurrent Network Training for Long or Event-based Sequences (2016, methodology): “Frame-based representation of an ‘8’ digit from the N-MNIST dataset [24] obtained by integrating all input spikes for each pixel.”
Opportunities for neuromorphic computing algorithms and applications (2022, neutral): “Shrestha et al. presented a hardware-friendly modification of the exponential STDP rule, albeit the classification performance on MNIST was lower than the best results achieved so far with SNNs 62 .”
Neuromorphic computing at scale (2025, neutral): “Several datasets have been developed specifically for benchmarking spiking systems 87–93 ; however, these are still not widely adopted 30 .”
EMNIST: Extending MNIST to handwritten letters (2017, background): “A good dataset needs to represent a sufﬁciently challenging problem to make it both useful and to ensure its longevity [2].”
HATS: Histograms of Averaged Time Surfaces for Robust Event-Based Object Classification (2018, methodology): “We validated our approach on ﬁve different datasets: four datasets generated by converting standard frame-based datasets to events (namely, the N-MNIST [48], N-Caltech101 [48], MNIST-DVS [63] and CIFAR10-DVS [35] datasets) and a novel dataset, recorded from real-world scenes and introduced for the ﬁ”
Deep Learning in Spiking Neural Networks (2018, neutral): “, 1998a,b), N-MNIST (Orchard et al., 2015), CIFAR-10, and”
High Speed and High Dynamic Range Video with an Event Camera (2019, methodology): “The N-MNIST (Neuromorphic-MNIST) and N-Caltech101 datasets [49] are eventbased versions of the MNIST [50] and Caltech101 [51] datasets.”
v2e: From Video Frames to Realistic DVS Events (2021, methodology): “N-Caltech 101 [23] is an event-based object recognition dataset generated from the Caltech 101 object recognition dataset [16].”
ESIM: an Open Event Camera Simulator (2018, background): “Most of these datasets target the task of object recognition [16, 17, 18], or gesture recognition [19].”
Synaptic Plasticity Dynamics for Deep Continuous Local Learning (DECOLLE) (2018, methodology): “The N-MNIST dataset was recorded with a Dynamic Vision Sensor (DVS) [30] mounted on a pan-tilt unit performing microsaccadic motions in front of a screen displaying samples from the MNIST dataset [39].”
Skimming Digits: Neuromorphic Classification of Spike-Encoded Images (2016, methodology): “This paper proposes and investigates four approaches to determining the output in a multi-class problem using SKIM, primarily applied to the N-MNIST dataset and also applicable to the multi-class classiﬁcation problems in the N-Caltech101 dataset introduced in Orchard et al. (2015a).”