Retinomorphic Event-Based Vision Sensors: Bioinspired Cameras With Spiking Output

By Christoph Posch, T. Serrano‐Gotarredona, B. Linares‐Barranco, Tobi Delbruck. 2014

428 citations recorded by the source corpus.

State-of-the-art image sensors suffer from significant limitations imposed by their very principle of operation. These sensors acquire the visual information as a series of “snapshot” images, recorded at discrete points in time. Visual information gets time quantized at a predetermined frame rate which has no relation to the dynamics present in the scene. Furthermore, each recorded frame conveys the information from all pixels, regardless of whether this information, or a part of it, has changed since the last frame had been acquired. This acquisition method limits the temporal resolution, potentially missing important information, and leads to redundancy in the recorded image data, unnecessarily inflating data rate and volume. Biology is leading the way to a more efficient style of image acquisition. Biological vision systems are driven by events happening within the scene in view, and not, like image sensors, by artificially created timing and control signals. Translating the frameless paradigm of biological vision to artificial imaging systems implies that control over the acquisition of visual information is no longer being imposed externally to an array of pixels but the decision making is transferred to the single pixel that handles its own information individually. In this paper, recent developments in bioinspired, neuromorphic optical sensing and artificial vision are presented and discussed. It is suggested that bioinspired vision systems have the potential to outperform conventional, frame-based vision systems in many application fields and to establish new benchmarks in terms of redundancy suppression and data compression, dynamic range, temporal resolution, and power efficiency. Demanding vision tasks such as real-time 3-D mapping, complex multiobject tracking, or fast visual feedback loops for sensory-motor action, tasks that often pose severe, sometimes insurmountable, challenges to conventional artificial vision systems, are in reach using bioinspired vision sensing and processing techniques.

Subfields: Cellular and Molecular Neuroscience, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cellular and Molecular Neuroscience

In Electrical and Electronic Engineering

Citations in the corpus

Phased LSTM: Accelerating Recurrent Network Training for Long or Event-based Sequences (2016, background): “The recordings come from an event-based vision sensor that is sensitive to local temporal contrast changes [26].”
HOTS: A Hierarchy of Event-Based Time-Surfaces for Pattern Recognition (2017, background): “A major drawback of frame-based acquisition is that it acquires information in a way that is independent of the dynamics of the underlying scene [3].”
Skimming Digits: Neuromorphic Classification of Spike-Encoded Images (2016, methodology): “A full treatment and in-depth review of these silicon retinae can be found in Posch et al. (2014) and the device used in this paper makes use of the Asynchronous Time-Based Imaging Sensor (ATIS) described in Posch et al. (2011).”
Going Deeper in Spiking Neural Networks: VGG and Residual Architectures (2018, background): “, 2015) or interfacing with silicon cochleas or event-driven sensors (Posch et al., 2014; 2011).”
Advancing Neuromorphic Computing With Loihi: A Survey of Results and Outlook (2021, background): “They exhibit an impressive combination of properties, including self-adaptation, low-power consumption, low-latency, and high dynamic range [100].”
DDD17: End-To-End DAVIS Driving Dataset (2017, methodology): “Rather than providing frame-based video as output, the event camera dynamic vision sensor (DVS) detects local changes in the brightness of individual pixels and asynchronously outputs those changes at the time of the changes (Lichtsteiner et al., 2008; Posch et al., 2014).”
v2e: From Video Frames to Realistic DVS Events (2021, background): “Since the ﬁrst DVS cameras, subsequent generations of DVS-type event cameras have been developed; see [6, 18, 25, 7] for surveys.”
Converting Static Image Datasets to Spiking Neuromorphic Datasets Using Saccades (2015, background): “Neuromorphic Vision sensors are specifically designed such that each pixel responds only to changes in pixel intensity (Posch et al., 2014).”
HATS: Histograms of Averaged Time Surfaces for Robust Event-Based Object Classification (2018, neutral): “This paper focuses on the problem of object classification using the output of a neuromorphic asynchronous event-based camera [15, 14, 53].”
Event-Based Vision: A Survey (2019, methodology): “to the part of the transient pathway(s) up to retinal ganglion cells. Similarly, the grayscale events of the ATIS correspond to the “sustained” or “what” pathway through the parvo layers of the brain [36], [42]. Event processing by SNNs: Artiﬁcial neurons, such as Leaky-Integrate and Fire or Adaptiv”
Conversion of Continuous-Valued Deep Networks to Efficient Event-Driven Networks for Image Classification (2017, background): “SNNs are also naturally suited to process input from event-based sensors (Posch et al., 2014; Liu et al., 2015), but even in classical framebased machine vision applications such as object recognition or detection, they have been shown to be accurate, fast, and efficient, in particular when…”
Learning to Detect Objects with a 1 Megapixel Event Camera (2020, background): “Event cameras [1, 2, 3, 4] promise a paradigm shift in computer vision by representing visual information in a fundamentally different way.”
All-in-one two-dimensional retinomorphic hardware device for motion detection and recognition (2021, background): “Vision sensors without memory capacity cannot extract and process motion information and are still not true retinomorphic hardware 14 .”