Keyframe-Based Visual-Inertial SLAM using Nonlinear Optimization

By Stefan Leutenegger, Paul Timothy Furgale, Vincent Rabaud, Margarita Chli, Kurt Konolige, Roland Yves Siegwart. Robotics: Science and Systems Conference, 2013

374 citations recorded by the source corpus.

The fusion of visual and inertial cues has become popular in robotics due to the complementary nature of the two sensing modalities. While most fusion strategies to date rely on filtering schemes, the visual robotics community has recently turned to non-linear optimization approaches for tasks such as visual Simultaneous Localization And Mapping (SLAM), following the discovery that this comes with significant advantages in quality of performance and computational complexity. Following this trend, we present a novel approach to tightly integrate visual measurements with readings from an Inertial Measurement Unit (IMU) in SLAM. An IMU error term is integrated with the landmark reprojection error in a fully probabilistic manner, resulting to a joint non-linear cost function to be optimized. Employing the powerful concept of ‘keyframes’ we partially marginalize old states to maintain a bounded-sized optimization window, ensuring real-time operation. Comparing against both vision-only and loosely-coupled visual-inertial algorithms, our experiments confirm the benefits of tight fusion in terms of accuracy and robustness.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/1678c758a1055a31b6a0167c42886b070a790471

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Robust state estimation for Micro Aerial Vehicles based on system dynamics (2015, methodology): “Since we only require the velocity, an onboard visual odometry system as for instance presented in [12], [13] would work as well.”
High Speed and High Dynamic Range Video with an Event Camera (2019, methodology): “the internal representation) plus optional images from a conventional camera, and fusing the tracks with inertial measurements using an existing optimization backend [53].”
Keyframe-based visual–inertial odometry using nonlinear optimization (2015, methodology): “In this article, which extends our previous work (Leutenegger et al., 2013), we propose a method that respects the aforementioned findings: we advocate tightly coupled fusion for best exploitation of all measurements and nonlinear optimization where possible rather than filtering, in order to…”
Visual-Inertial Navigation Algorithm Development Using Photorealistic Camera Simulation in the Loop (2018, background): “The literature on visual inertial navigation is vast, including approaches based on ﬁltering, e.g., [19], [20], ﬁxed-lag smoothing, e.g., [21], [22], and full smoothing [23]–[25].”
Large-scale direct SLAM for omnidirectional cameras (2015, neutral): “This can be done in a filtering framework [3], [4], [5], or in a keyframe-based non-linear optimization framework [6], [7], [8].”
Events-To-Video: Bringing Modern Computer Vision to Event Cameras (2019, methodology): “The state-of-the-art EVIO system, UltimateSLAM [41], operates by independently tracking visual features from pseudo-images reconstructed from events and optional images from a conventional camera, and fusing the tracks with inertial measurements using an existing optimization backend [23].”
Robust visual inertial odometry using a direct EKF-based approach (2015, result): “[16] describe a tightly coupled approach in which the robot trajectory and sparse 3D landmarks are estimated in a joint optimization problem using inertial error terms as well as the reprojection error of the landmarks in the camera image.”
Kimera: an Open-Source Library for Real-Time Metric-Semantic Localization and Mapping (2019, methodology): “Table II compares the Root Mean Squared Error (RMSE) of the Absolute Translation Error (ATE) of KimeraVIO against state-of-the-art open-source VIO pipelines: OKVIS [73], MSCKF [74], ROVIO [75], VINS-Mono [24], and SVO-GTSAM [76] using the independently reported values in [77] and the self-reported v”
Get Out of My Lab: Large-scale, Real-Time Visual-Inertial Localization (2015, methodology): “At the same time, the proper marginalization of past measurements [12, 45, 35, 24] is key to obtain a smooth pose estimate.”
Ultimate SLAM? Combining Events, Images, and IMU for Robust Visual SLAM in HDR and High-Speed Scenarios (2017, methodology): “For details, we refer the reader to [2].”
Event-Based Vision: A Survey (2019, methodology): “1), and then these point trajectories on the image plane are fused with IMU data using modern VIO algorithms, such as [224], [225], [226].”
Kimera: From SLAM to spatial perception with 3D dynamic scene graphs (2021, methodology): “Table 1 compares the root mean squared error (RMSE) of the absolute translation error (ATE) of Kimera-VIO against state-of-the-art open-source VIO pipelines: OKVIS (Leutenegger et al., 2013), MSCKF (Mourikis and Roumeliotis, 2007), ROVIO (Bloesch et al.”
Visual SLAM and Structure from Motion in Dynamic Environments (2018, background): “Other approaches leverage external sensors such as an inertial measurement unit (IMU) to solve this problem [67, 92] by estimating the camera ego-motion.”
A Benchmark Comparison of Monocular Visual-Inertial Odometry Algorithms for Flying Robots (2018, methodology): “• OKVIS [2] - a keyframe- and optimization-based sliding window estimator using landmark reprojection errors.”
Factor Graphs for Robot Perception (2017, background): “[131, 132] also present VIO and SLAM algorithms that are visualized as factor graphs.”
Computer Vision for Autonomous Vehicles: Problems, Datasets and State-of-the-Art (2017, methodology): “Joint Graph Decomposition and Node Labeling – Levinkov et al. (2016) 9.8 23.2 16.8 20.3 Pixel-level Encoding for Instance Segmentation – Uhrig et al. (2016) 8.”
ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual–Inertial, and Multimap SLAM (2020, methodology): “The ﬁrst tightly coupled visual odometry system based on keyframes and bundle adjustment was OKVIS [38], [39] that is also able to use monocular and stereo vision.”
LINS: A Lidar-Inertial State Estimator for Robust and Efficient Navigation (2019, neutral): “Tightly-coupled approaches directly fuse lidar and inertial measurements through joint-optimization, which can be categorized into optimization based [16], [17] and extended Kalman filter (EKF) based [18], [15].”