Hydra: A Real-time Spatial Perception System for 3D Scene Graph Construction and Optimization

By Nathan Hughes, Yun Chang, Luca Carlone. Robotics: Science and Systems Conference, 2022

325 citations recorded by the source corpus.

3D scene graphs have recently emerged as a powerful high-level representation of 3D environments. A 3D scene graph describes the environment as a layered graph where nodes represent spatial concepts at multiple levels of abstraction and edges represent relations between concepts. While 3D scene graphs can serve as an advanced"mental model"for robots, how to build such a rich representation in real-time is still uncharted territory. This paper describes a real-time Spatial Perception System, a suite of algorithms to build a 3D scene graph from sensor data in real-time. Our first contribution is to develop real-time algorithms to incrementally construct the layers of a scene graph as the robot explores the environment; these algorithms build a local Euclidean Signed Distance Function (ESDF) around the current robot location, extract a topological map of places from the ESDF, and then segment the places into rooms using an approach inspired by community-detection techniques. Our second contribution is to investigate loop closure detection and optimization in 3D scene graphs. We show that 3D scene graphs allow defining hierarchical descriptors for loop closure detection; our descriptors capture statistics across layers in the scene graph, ranging from low-level visual appearance to summary statistics about objects and places. We then propose the first algorithm to optimize a 3D scene graph in response to loop closures; our approach relies on embedded deformation graphs to simultaneously correct all layers of the scene graph. We implement the proposed Spatial Perception System into a architecture named Hydra, that combines fast early and mid-level perception processes with slower high-level perception. We evaluate Hydra on simulated and real data and show it is able to reconstruct 3D scene graphs with an accuracy comparable with batch offline methods despite running online.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/104fc0dbe5a3fdd81f3a7e372e09147e0ce9c890

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

A Survey on Active Simultaneous Localization and Mapping: State of the Art and New Frontiers (2022, background): “, allowing for novel loop closure detection methods [242]); 3) hierarchical representations may enable novel and more computationally efficient planning methods.”
Text2Motion: from natural language instructions to feasible plans (2023, methodology): “Suitable examples could be scene graph generation methods [43, 27] or forms of text-based scene description [22, 35, 58].”
Visual Language Maps for Robot Navigation (2022, background): “Although progress has been made at generating more abstract maps, such as scene graphs [20], [21], current approaches are limited to a predefined set of semantic classes.”
eMEM: A Hybrid Spatio-Temporal Memory System For Embodied Agents (2026, unknown): “Hydra [Hughes et al., 2022] builds hierarchical 3D scene graphs in real time from sensor data. eMEM does not replicate this geometric scene graph; it instead operates at a higher level, storing what the agent remembers about places rather than the current geometric state.”