Ultimate SLAM? Combining Events, Images, and IMU for Robust Visual SLAM in HDR and High-Speed Scenarios

By A. Vidal, Henri Rebecq, Timo Horstschaefer, Davide Scaramuzza. IEEE Robotics and Automation Letters, 2017

534 citations recorded by the source corpus.

Event cameras are bioinspired vision sensors that output pixel-level brightness changes instead of standard intensity frames. These cameras do not suffer from motion blur and have a very high dynamic range, which enables them to provide reliable visual information during high-speed motions or in scenes characterized by high dynamic range. However, event cameras output only little information when the amount of motion is limited, such as in the case of almost still motion. Conversely, standard cameras provide instant and rich information about the environment most of the time (in low-speed and good lighting scenarios), but they fail severely in case of fast motions, or difficult lighting such as high dynamic range or low light scenes. In this letter, we present the first state estimation pipeline that leverages the complementary advantages of these two sensors by fusing in a tightly coupled manner events, standard frames, and inertial measurements. We show on the publicly available Event Camera Dataset that our hybrid pipeline leads to an accuracy improvement of 130% over event-only pipelines, and 85% over standard-frames-only visual-inertial systems, while still being computationally tractable. Furthermore, we use our pipeline to demonstrate—to the best of our knowledge—the first autonomous quadrotor flight using an event camera for state estimation, unlocking flight scenarios that were not reachable with traditional visual-inertial odometry, such as low-light environments and high dynamic range scenes. Videos of the experiments: http://rpg.ifi.uzh.ch/ultimateslam.html

Subfields: Artificial Intelligence, Electrical and Electronic Engineering

View Paper: https://www.semanticscholar.org/paper/773f448639766fa98e71e733fab5564466acfafa

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

Citations in the corpus

High Speed and High Dynamic Range Video with an Event Camera (2019, methodology): “Second, exploiting the sparsity of the TABLE 4 Mean Translation Error (in Meters) on the Sequences from [38] Inputs Ours E+I U.SLAM E+I U.SLAM E+F+I HF E+I MR E+I VINS-Mono F+I shapes_translation 0.18 0.32 0.17 failed 2.00 0.93 poster_translation 0.05 0.09 0.06 0.49 0.15 failed boxes_translation 0.1”
Reducto: On-Camera Filtering for Resource-Efficient Real-Time Video Analytics (2020, background): “The computer vision (CV) community [20, 24, 44, 46, 59, 60, 66, 68, 82, 83] has discovered a slew of low-level video features that extract frame differences [21], such as Edge and Pixel .”
ESIM: an Open Event Camera Simulator (2018, background): “…HDR, no motion blur), event cameras promise to unlock robust and high-speed perception in situations that are currently not accessible to standard cameras: for example, balancing a pen [2], tracking features in the blind time between two frames [3], or ﬂying a quadcopter in a dark room [4].”
End-to-End Learning of Representations for Asynchronous Event-Based Data (2019, methodology): “Despite their differences, both paradigms have been successfully applied on various vision tasks, including tracking [19,21,40,42], depth estimation [3,52,67], visual odometry [27, 54, 57, 66], recognition [29, 44], and optical ﬂow estimation [7, 69].”
BAD SLAM: Bundle Adjusted Direct RGB-D SLAM (2019, background): “This for example enables augmented reality [32] and perception for self-driving cars [64].”
Events-To-Video: Bringing Modern Computer Vision to Event Cameras (2019, methodology): “As is standard [53, 38, 41], we use sequences from the Event Camera Dataset [30], which contain events, frames, and IMU measurements from a DAVIS240C [7] sensor.”
A Unifying Contrast Maximization Framework for Event Cameras, with Applications to Motion, Depth, and Optical Flow Estimation (2018, methodology): “These motion-corrected image patches can be useful for feature tracking, such as [27, 29].”
Deep Learning With Spiking Neurons: Opportunities and Challenges (2018, background): “…et al., 2018) and robotics (Mueggler et al., 2017) have been released, and together with convincing results on problems where conventional systems struggle (Kim et al., 2016; Vidal et al., 2018), we expect that this will lead to increasing demand for eﬃcient event-based post-processing systems.”
Event-Based Vision: A Survey (2019, methodology): “This could be [27].”