SpatialRGPT: Grounded Spatial Reasoning in Vision Language Model
By An-Chieh Cheng, Hongxu Yin, Yang Fu, Qiushan Guo, Ruihan Yang, Jan Kautz, Xiaolong Wang, Sifei Liu. Neural Information Processing Systems, 2024
364 citations recorded by the source corpus.
Vision Language Models (VLMs) have demonstrated remarkable performance in 2D vision and language tasks. However, their ability to reason about spatial arrangements remains limited. In this work, we introduce Spatial Region GPT (SpatialRGPT) to enhance VLMs' spatial perception and reasoning capabilities. SpatialRGPT advances VLMs' spatial understanding through two key innovations: (1) a data curation pipeline that enables effective learning of regional representation from 3D scene graphs, and (2) a flexible plugin module for integrating depth information into the visual encoder of existing VLMs. During inference, when provided with user-specified region proposals, SpatialRGPT can accurately perceive their relative directions and distances. Additionally, we propose SpatialRGBT-Bench, a benchmark with ground-truth 3D annotations encompassing indoor, outdoor, and simulated environments, for evaluating 3D spatial cognition in VLMs. Our results demonstrate that SpatialRGPT significantly enhances performance in spatial reasoning tasks, both with and without local region prompts. The model also exhibits strong generalization capabilities, effectively reasoning about complex spatial relations and functioning as a region-aware dense reward annotator for robotic tasks. Code, dataset, and benchmark are released at https://www.anjiecheng.me/SpatialRGPT
Subfields: Computer Networks and Communications, Computer Vision and Pattern Recognition, Geography, Planning and Development
View Paper: https://www.semanticscholar.org/paper/0072019b934df4a802c2f948ed090c7745b32dd5
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Networks and Communications
In Computer Vision and Pattern Recognition
Citations in the corpus
- Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces (2024, positive): “However, grounding MLLMs in the real world presents significant challenges for models’ visual-spatial intelligence, motivating recent efforts [10, 13, 16, 29, 41, 48, 94, 107].”