Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces

By Jihan Yang, Shusheng Yang, Anjali Gupta, Rilyn Han, Fei-Fei Li, Saining Xie. Computer Vision and Pattern Recognition, 2024

669 citations recorded by the source corpus.

Humans possess the visual-spatial intelligence to remember spaces from sequential visual observations. However, can Multimodal Large Language Models (MLLMs) trained on million-scale video datasets also "think in space" from videos? We present a novel video-based visual-spatial intelligence benchmark (VSI-Bench) of over 5,000 question-answer pairs, and find that MLLMs exhibit competitive—though subhuman—visual-spatial intelligence. We probe models to express how they think in space both linguistically and visually and find that while spatial reasoning capabilities remain the primary bottleneck for MLLMs to reach higher benchmark performance, local world models and spatial awareness do emerge within these models. Notably, prevailing linguistic reasoning techniques (e.g., chain-of-thought, self-consistency, tree-of-thoughts) fail to improve performance, whereas explicitly generating cognitive maps during question-answering enhances MLLMs’ spatial distance ability.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/376461a2c049f6fa51a4303853fdc672e4d07a0d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Tong Lu
2. Wenhai Wang
3. Jianfeng Gao
4. Jingren Zhou
5. Xizhou Zhu
6. Dahua Lin
7. Lewei Lu
8. Zhaoyang Liu

Citations in the corpus

• Video-R1: Reinforcing Video Reasoning in MLLMs (2025, neutral): “Our experiments show that Video-R1 achieves consistent and significant improvements across a suite of challenging video reasoning benchmarks, including VSI-Bench [24], VideoMMMU [8], MMVU [32], MVBench [13], TempCompass [18], and VideoMME [4].”
• InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency (2025, positive): “MMLU-Pro [61], GAOKAO [177], IFEval [185]; (4) Agentic Tasks : SGP-Bench [102], ScreenSpot [16], ScreenSpot-v2 [150], OSWorld-G [156], VSI-Bench [161], ERQA [121], SpaCE-10 [38].”
• SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Model (2025, neutral): “…and lack precise perception and comprehension of the 3D physical world — where humans instinctively construct rich, structured mental representations of space, effortlessly aligning objects within a canonical, intuitive, and even personally tailored workspace for manipulation [20, 40, 52, 63, 67].”
• Kimi-VL Technical Report (2025, positive): “VSI-Bench (Yang et al. 2024) is designed to evaluate the visual-spatial comprehensive capabilities of large multimodal models.”
• Qwen3-VL Technical Report (2025, positive): “…understanding, Qwen3-VL’s performance is rigorously benchmarked against leading SOTA models using a challenging suite of benchmarks: ERQA (Team et al., 2025), VSIBench (Yang et al., 2025b), EmbSpatial (Du et al., 2024), RefSpatial (Zhou et al., 2025), and RoboSpatialHome (Song et al., 2025a).”
• InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models (2025, positive): “Table 10 reports the performance results on the Visual-Spatial Intelligence Benchmark (VSI-Bench) [134], where InternVL3 is compared against other state-of-the-art MLLMs.”