Learning 3D Semantic Scene Graphs From 3D Indoor Reconstructions

By Johanna Wald, Helisa Dhamo, Nassir Navab, Federico Tombari. 2020

331 citations recorded by the source corpus.

Scene understanding has been of high interest in computer vision. It encompasses not only identifying objects in a scene, but also their relationships within the given context. With this goal, a recent line of works tackles 3D semantic segmentation and scene layout prediction. In our work we focus on scene graphs, a data structure that organizes the entities of a scene in a graph, where objects are nodes and their relationships modeled as edges. We leverage inference on scene graphs as a way to carry out 3D scene understanding, mapping objects and their relationships. In particular, we propose a learned method that regresses a scene graph from the point cloud of a scene. Our novel architecture is based on PointNet and Graph Convolutional Networks (GCN). In addition, we introduce 3DSSG, a semiautomatically generated dataset, that contains semantically rich scene graphs of 3D scenes. We show the application of our method in a domain-agnostic retrieval task, where graphs serve as an intermediate representation for 3D-3D and 2D-3D matching.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

Citations in the corpus

Continuous Scene Representations for Embodied AI (2022, background): “However, others propose to create 3D scene graphs that encode 3D relationships between the components of a scene [2, 12, 20, 51, 66].”
SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities (2024, methodology): “When applying these spatial concepts to reasoning, prior works often focus on explicit spatial scene memories [26, 27] or spatial scene graphs [29, 30, 59, 60].”
Embodied Semantic Scene Graph Generation (2021, methodology): “It is widely used in tasks that utilize the scene graph to bridge the gap between semantic and visual information [26][27].”
A Comprehensive Survey of Scene Graphs: Generation and Application (2021, methodology): “[7] and 3DSSG [172] are two large-scale 3D semantic scene graphs containing indoor architecture or 3D reconstructions of real scenes.”
Kimera: From SLAM to spatial perception with 3D dynamic scene graphs (2021, background): “Recently, Wald et al. (2020) propose to learn from point clouds a 3D semantic scene graph that focuses on representing semantically meaningful inter-instance relationships.”
3D-FRONT: 3D Furnished Rooms with layOuts and semaNTics (2020, background): “Over the past years, a large number of RGB-D benchmarks have been constructed and made publicly available [21, 3, 27, 36, 43, 33, 19, 6, 9, 18, 26, 23, 49, 24, 4, 32, 35, 42, 11, 17, 47, 38, 2].”
OpenScene: 3D Scene Understanding with Open Vocabularies (2022, unknown): “Note that for indoor datasets like ScanNet and Matterport3D where the depth images are provided, we also conduct occlusion tests to guarantee the pixel u are only paired with a visible surface point p .”
SpatialRGPT: Grounded Spatial Reasoning in Vision Language Model (2024, background): “For the person in Region [1], Region [3] is in the 10 o'clock direction.”
ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning (2023, background): “3D scene graphs (3DSGs) address the second limitation by compactly and efficiently describing scenes with graph structures, with nodes representing objects and edges encoding inter-object relationships [44]– [48].”