Event-Based Vision Meets Deep Learning on Steering Prediction for Self-Driving Cars

By Ana I. Maqueda, Antonio Loquercio, Guillermo Gallego, Narciso García, Davide Scaramuzza. 2018

616 citations recorded by the source corpus.

Event cameras are bio-inspired vision sensors that naturally capture the dynamics of a scene, filtering out redundant information. This paper presents a deep neural network approach that unlocks the potential of event cameras on a challenging motion-estimation task: prediction of a vehicle's steering angle. To make the best out of this sensor-algorithm combination, we adapt state-of-the-art convolutional architectures to the output of event sensors and extensively evaluate the performance of our approach on a publicly available large scale event-camera dataset (~1000 km). We present qualitative and quantitative explanations of why event cameras allow robust steering prediction even in cases where traditional cameras fail, e.g. challenging illumination conditions and fast motion. Finally, we demonstrate the advantages of leveraging transfer learning from traditional to event-based vision, and show that our approach outperforms state-of-the-art algorithms based on standard cameras.

Subfields: Cognitive Neuroscience, Computer Vision and Pattern Recognition, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Computer Vision and Pattern Recognition

In Electrical and Electronic Engineering

Citations in the corpus

A Survey of Autonomous Driving: Common Practices and Emerging Technologies (2019, background): “types gaining interest for ADS include flash cameras [77], thermal cameras [79], [80], and event cameras [78].”
Event-Based Vision: A Survey (2019, methodology): “Sample applications include classiﬁcation [134], [135], image reconstruction [8], [102], steering angle prediction [91], [136], and estimation of optical ﬂow [22], [103], [137], depth [137] or ego-motion [103].”
End-to-End Learning of Driving Models with Surround-View Cameras and Route Planners (2018, background): “Autonomous driving has seen dramatic advances in recent years, for instance for road scene parsing [1,2,3,4], lane following [5,6,7], path planning [8,9,10,11], and end-toend driving models [12,13,14,15].”
Learning Task-Oriented Communication for Edge Inference: An Information Bottleneck Approach (2021, background): “However, the transmission latency remains a bottleneck for applications with stringent delay requirements [15], [16], [17], as a huge volume of data (e.”
Events-To-Video: Bringing Modern Computer Vision to Event Cameras (2019, background): “Examples are the integration of events on the image plane [39, 26, 27] as well as time surfaces [21, 45, 55, 51, 55].”
End-to-End Learning of Representations for Asynchronous Event-Based Data (2019, background): “Recently, several works have proposed to use standard learning architectures as an alternative to SNNs [36, 41, 59, 68, 69].”
EKLT: Asynchronous Photometric Feature Tracking Using Events and Frames (2018, background): “2018, 2017) rather than over a fixed time τ (Maqueda et al. 2018; Bardow et al. 2016).”
A Unifying Contrast Maximization Framework for Event Cameras, with Applications to Motion, Depth, and Optical Flow Estimation (2018, methodology): “…temporal information of the events and accumulate them into frames to re-utilize traditional, image-based computer vision algorithms [16, 17, 18], and 2b) methods that exploit the temporal information of the events during the estimation process, and therefore cannot re-utilize traditional…”
Self-Balancing Federated Learning With Global Imbalanced Data in Mobile Systems (2021, background): “WITH the enormous success of deep learning (DL) [2], more and more neural network applications enter every aspect of our lives, such as self-driving cars [3], face [4] and speech recognition [5].”