3D Scene Graph: A Structure for Unified Semantics, 3D Space, and Camera

By Iro Armeni, Zhi-Yang He, JunYoung Gwak, Amir Zamir, Martin Fischer, Jitendra Malik, Silvio Savarese. 2019

531 citations recorded by the source corpus.

A comprehensive semantic understanding of a scene is important for many applications - but in what space should diverse semantic information (e.g., objects, scene categories, material types, 3D shapes, etc.) be grounded and what should be its structure? Aspiring to have one unified structure that hosts diverse types of semantics, we follow the Scene Graph paradigm in 3D, generating a 3D Scene Graph. Given a 3D mesh and registered panoramic images, we construct a graph that spans the entire building and includes semantics on objects (e.g., class, material, shape and other attributes), rooms (e.g., function, illumination type, etc.) and cameras (e.g., location, etc.), as well as the relationships among these entities. However, this process is prohibitively labor heavy if done manually. To alleviate this we devise a semi-automatic framework that employs existing detection methods and enhances them using two main constraints: I. framing of query images sampled on panoramas to maximize the performance of 2D detectors, and II. multi-view consistency enforcement across 2D detections that originate in different camera locations.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Continuous Scene Representations for Embodied AI (2022, background): “Towards building such representations, scene graphs [2, 29, 65] are a candidate as they provide a compact and explicit description of a scene.”
ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objects (2020, background): “[18] provide object segmentations and labels on 35 homes.”
Continual Learning of Semantic Segmentation using Complementary 2D-3D Data Representations (2021, background): “Prior work focuses on accumulation of 2D semantic network predictions in a global 3D representation to enhance the semantic predictions [18, 12, 25, 3].”
Image Scene Graph Generation (SGG) Benchmark (2021, background): “…for building more visually-grounded and compositional models for other computer vision and vision-language tasks, like image/text retrieval [9, 17], image cap-tioning [4, 20, 6], visual question answering [3, 5], action recognition [7], image generation [8, 14], and 3D scene understanding [2].”
Learning to Generate Scene Graph from Natural Language Supervision (2021, background): “Scene graph has emerged as a structured representation for many vision tasks, including action recognition [19], 3D scene understanding [1, 49], image generation and editing [21, 13], and vision-language tasks (e.”
Open-vocabulary Queryable Scene Representations for Real World Planning (2022, background): “Recently, topological maps [17], [18] and scene graphs [19], [20] emerge as an effective discrete representation of scenes.”
Omnidata: A Scalable Pipeline for Making Multi-Task Mid-Level Vision Datasets from 3D Scans (2021, methodology): “The ranking of transfers depended on the task, which might be due to the sparse panoptic labels on Taskonomy (from the followup paper [2]), but we believe the dependency is true in general. can we do with such datasets?”
ZSON: Zero-Shot Object-Goal Navigation using Multimodal Goal Embeddings (2022, background): “This is in stark contrast to ObjectNav , which requires annotating 3D meshes [7–9] and potentially collecting large-scale human demonstrations [10] for training.”
Object Goal Navigation using Goal-Oriented Semantic Exploration (2020, unknown): “[2].”
Embodied Semantic Scene Graph Generation (2021, background): “Armeni et al. [10] presents a 3D scene graph, which uniﬁes multi-modal semantic information in the 3D space.”
Hierarchical Planning for Long-Horizon Manipulation with Geometric and Symbolic Scene Graphs (2020, methodology): “We propose to represent a manipulation scene as scene graphs [1, 28], a two-level abstraction shown in Figure 1.”
A Survey on Active Simultaneous Localization and Mapping: State of the Art and New Frontiers (2022, background): “Very recent work goes even further and develops spatial perception systems that infer hierarchical map representations, in the form of 3-D scene graphs [103], [241], [242].”
Hydra: A Real-time Spatial Perception System for 3D Scene Graph Construction and Optimization (2022, unknown): “[4] pioneered the use of 3D scene graphs in computer vision and proposed the first algorithms to parse a metric-semantic 3D mesh into a 3D scene graph.”
Learning 3D Semantic Scene Graphs From 3D Indoor Reconstructions (2020, background): “In contrast to previous works [19, 3, 4, 45], our nodes are not”
A Comprehensive Survey of Scene Graphs: Generation and Application (2021, background): “[7] and 3DSSG [172] are two large-scale 3D semantic scene graphs containing indoor architecture or 3D reconstructions of real scenes.”
Learning Object-conditioned Exploration using Distributed Soft Actor Critic (2020, methodology): “We use a subset of the worlds for which object segmentations and labels are available, as provided in [32].”
Kimera: From SLAM to spatial perception with 3D dynamic scene graphs (2021, methodology): “While not investigated in this paper, we refer the reader to (Armeni et al. 2019) for a more comprehensive list of attributes, including materials and affordances.”