3D Dynamic Scene Graphs: Actionable Spatial Perception with Places, Objects, and Humans

By Antoni Rosinol, Arjun Gupta, Marcus Abate, Jingnan Shi, Luca Carlone. 2020

271 citations recorded by the source corpus.

We present a unified representation for actionable spatial perception: 3D Dynamic Scene Graphs. Scene graphs are directed graphs where nodes represent entities in the scene (e.g. objects, walls, rooms), and edges represent relations (e.g. inclusion, adjacency) among nodes. Dynamic scene graphs (DSGs) extend this notion to represent dynamic scenes with moving agents (e.g. humans, robots), and to include actionable information that supports planning and decision-making (e.g. spatio-temporal relations, topology at different levels of abstraction). Our second contribution is to provide the first fully automatic Spatial PerceptIon eNgine(SPIN) to build a DSG from visual-inertial data. We integrate state-of-the-art techniques for object and human detection and pose estimation, and we describe how to robustly infer object, robot, and human nodes in crowded scenes. To the best of our knowledge, this is the first paper that reconciles visual-inertial SLAM and dense human mesh tracking. Moreover, we provide algorithms to obtain hierarchical representations of indoor environments (e.g. places, structures, rooms) and their relations. Our third contribution is to demonstrate the proposed spatial perception engine in a photo-realistic Unity-based simulator, where we assess its robustness and expressiveness. Finally, we discuss the implications of our proposal on modern robotics applications. 3D Dynamic Scene Graphs can have a profound impact on planning and decision-making, human-robot interaction, long-term autonomy, and scene prediction. A video abstract is available at this https URL

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Continuous Scene Representations for Embodied AI (2022, methodology): “Our method is closer to the approaches that create scene graphs in embodied or physics based settings [6, 17, 31, 39, 67].”
Hydra: A Real-time Spatial Perception System for 3D Scene Graph Construction and Optimization (2022, unknown): “Extending our prior works [49, 50] to operate in real-time is non-trivial.”
ConceptFusion: Open-set Multimodal 3D Mapping (2023, background): “One of the major catalysts of continued progress in 3D spatial perception [1, 2, 3, 4] has been the advent of deep convolutional networks trained on large datasets of images.”
Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representation (2022, background): “Similarly, Neural Scene Graphs (NSGs) [40] uses a separate conditional NeRF for each object category, and a multiplane neural representation for the background.”
DynaSLAM II: Tightly-Coupled Multi-Object Tracking and SLAM (2020, background): “There is a new work by Rosinol et al. [18] that reconciles visual-inertial SLAM and dense mesh tracking, focusing mostly on humans, that shows impressive results in simulation.”
Kimera-Multi: Robust, Distributed, Dense Metric-Semantic SLAM for Multi-Robot Systems (2021, background): “and meshes [16, 31, 5], or 3D scene graphs [30].”
Hierarchical Planning for Long-Horizon Manipulation with Geometric and Symbolic Scene Graphs (2020, methodology): “This has motivated a variety of graph-based scene representations [1, 19, 28, 29, 39] developed in the robotics and computer vision community.”
NeRF-SLAM: Real-Time Dense Monocular SLAM with Neural Radiance Fields (2022, methodology): “Beyond metric-semantic SLAM, our approach can be used as the mapping engine for high-level scene understanding, such as for building 3D Dynamic Scene Graphs [40], [41].”
Kimera: From SLAM to spatial perception with 3D dynamic scene graphs (2021, methodology): “KimeraCore (Rosinol et al. 2020a) is in charge of the real-time metric-semantic reconstruction of the scene, and comprises the following modules: • Kimera-VIO (Section 3.1) is a visual-inertial odometry (VIO) module for fast and locally accurate 3D pose estimation (localization).”
Embodied Semantic Scene Graph Generation (2021, background): “Recently, Rosinol et al. [11] proposes a 3D dynamic scene graph which extends to represent dynamic scenes with moving agents.”
Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wild (2020, background): “[55] is another recent approach that constructs a 3D scene graph of humans and objects from video and inertial data captured indoors.”