Kimera: From SLAM to spatial perception with 3D dynamic scene graphs

By Antoni Rosinol, Andrew Violette, Marcus Abate, Nathan Hughes, Yun Chang, Jingnan Shi, Arjun Gupta, Luca Carlone. 2021

364 citations recorded by the source corpus.

Humans are able to form a complex mental model of the environment they move in. This mental model captures geometric and semantic aspects of the scene, describes the environment at multiple levels of abstractions (e.g., objects, rooms, buildings), includes static and dynamic entities and their relations (e.g., a person is in a room at a given time). In contrast, current robots’ internal representations still provide a partial and fragmented understanding of the environment, either in the form of a sparse or dense set of geometric primitives (e.g., points, lines, planes, and voxels), or as a collection of objects. This article attempts to reduce the gap between robot and human perception by introducing a novel representation, a 3D dynamic scene graph (DSG), that seamlessly captures metric and semantic aspects of a dynamic environment. A DSG is a layered graph where nodes represent spatial concepts at different levels of abstraction, and edges represent spatiotemporal relations among nodes. Our second contribution is Kimera, the first fully automatic method to build a DSG from visual–inertial data. Kimera includes accurate algorithms for visual–inertial simultaneous localization and mapping (SLAM), metric–semantic 3D reconstruction, object localization, human pose and shape estimation, and scene parsing. Our third contribution is a comprehensive evaluation of Kimera in real-life datasets and photo-realistic simulations, including a newly released dataset, uHumans2, which simulates a collection of crowded indoor and outdoor scenes. Our evaluation shows that Kimera achieves competitive performance in visual–inertial SLAM, estimates an accurate 3D metric–semantic mesh model in real-time, and builds a DSG of a complex indoor environment with tens of objects and humans in minutes. Our final contribution is to showcase how to use a DSG for real-time hierarchical semantic path-planning. The core modules in Kimera have been released open source.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Geology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In Geology

Citations in the corpus

Continuous Scene Representations for Embodied AI (2022, background): “Recently, approaches have been designed to handle more complex dynamic scenes [40,55].”
A Survey on Active Simultaneous Localization and Mapping: State of the Art and New Frontiers (2022, background): “[103] combined metric, semantic, and topological representations into a single model, a 3-D scene graph.”
NeRF-SLAM: Real-Time Dense Monocular SLAM with Neural Radiance Fields (2022, background): “Future work can leverage our approach to extend the definition of metric-semantic SLAM [39], which typically only considers geometric and semantic properties, by building representations that are also photometrically accurate.”
Text2Motion: from natural language instructions to feasible plans (2023, methodology): “Suitable examples could be scene graph generation methods [43, 27] or forms of text-based scene description [22, 35, 58].”
Kimera-Multi: Robust, Distributed, Dense Metric-Semantic SLAM for Multi-Robot Systems (2021, background): “Recent work includes systems that can build metric-semantic 3D models in real-time using a multi-core CPU, including Kimera [1] and Voxblox++ [5].”
A Comprehensive Survey of Scene Graphs: Generation and Application (2021, background): “Similarly, 3DSSG (3D Semantic Scene Graphs) [172] and 3D-DSG (3D Dynamic Scene Graphs) [13] also studied the scene understanding of indoor 3D environments.”
ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning (2023, background): “These approaches have enabled building real-time systems that can dynamically build up hierarchical 3D scene representations [49]–[51], and more recently shown that various robotics planning tasks can benefit from efficiency and compactness of 3DSGs [52], [53].”
Hydra: A Real-time Spatial Perception System for 3D Scene Graph Construction and Optimization (2022, positive): “Extending our prior works [49, 50] to operate in real-time is non-trivial.”