ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning

By Qiao Gu, Ali Kuwajerwala, Sacha Morin, Krishna Murthy Jatavallabhula, Bipasha Sen, Aditya Agarwal, Corban Rivera, William Paul, Kirsty Ellis, Ramalingam Chellappa and 6 more. IEEE International Conference on Robotics and Automation, 2023

508 citations recorded by the source corpus.

For robots to perform a wide variety of tasks, they require a 3D representation of the world that is semantically rich, yet compact and efficient for task-driven perception and planning. Recent approaches have attempted to leverage features from large vision-language models to encode semantics in 3D representations. However, these approaches tend to produce maps with per-point feature vectors, which do not scale well in larger environments, nor do they contain semantic spatial relationships between entities in the environment, which are useful for downstream planning. In this work, we propose ConceptGraphs, an open-vocabulary graph-structured representation for 3D scenes. ConceptGraphs is built by leveraging 2D foundation models and fusing their output to 3D by multi-view association. The resulting representations generalize to novel semantic classes, without the need to collect large 3D datasets or finetune models. We demonstrate the utility of this representation through a number of downstream planning tasks that are specified through abstract (language) prompts and require complex reasoning over spatial and semantic concepts. To explore the full scope of our experiments and results, we encourage readers to visit our project webpage.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/174a3290ff0040e0f6a7a0b43bcd752c456350a0

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

1. Jan Kautz

In Computer Vision and Pattern Recognition

1. Hongxu Yin
2. Ruihan Yang
3. Yang Fu
4. Jan Kautz
5. Xiaolong Wang
6. Sifei Liu
7. Qiushan Guo
8. An-Chieh Cheng

Citations in the corpus

• A Survey on Vision--Language--Action Models for Embodied AI (2024, unknown): “ConceptGraphs [170] introduces a method to convert observation sequences into open-vocabulary 3D scene graphs.”
• OpenEQA: Embodied Question Answering in the Era of Foundation Models (2024, unknown): “However, we note that these paths are not necessarily optimal, and thus E > 100% is theoretically possible.”
• Aligning Cyber Space With Physical World: A Comprehensive Survey on Embodied AI (2024, unknown): “Similarly, ConceptGraphs [189] also adopts 3-”
• SpatialRGPT: Grounded Spatial Reasoning in Vision Language Model (2024, methodology): “ConceptGraph [19] avoids directly incorporating 3D representations into LLMs.”
• eMEM: A Hybrid Spatio-Temporal Memory System For Embodied Agents (2026, unknown): “The memory survey of Zhang et al. [2024] provides a taxonomy onto which eMEM’s architecture maps cleanly.”