Visual Spatial Reasoning

By Fangyu Liu, Guy Edward Toh Emerson, Nigel Collier. Transactions of the Association for Computational Linguistics, 2022

352 citations recorded by the source corpus.

Spatial relations are a basic part of human cognition. However, they are expressed in natural language in a variety of ways, and previous work has suggested that current vision-and-language models (VLMs) struggle to capture relational information. In this paper, we present Visual Spatial Reasoning (VSR), a dataset containing more than 10k natural text-image pairs with 66 types of spatial relations in English (e.g., under, in front of, facing). While using a seemingly simple annotation format, we show how the dataset includes challenging linguistic phenomena, such as varying reference frames. We demonstrate a large gap between human and model performance: The human ceiling is above 95%, while state-of-the-art models only achieve around 70%. We observe that VLMs’ by-relation performances have little correlation with the number of training examples and the tested models are in general incapable of recognising relations concerning the orientations of objects.1

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/354b48677e314ef2f47512c5a81723cfd17dd05d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning (2023, background): “…11 task categories and 26 datasets, including image captioning [10, 11, 12], image captioning with reading comprehension [13], visual reasoning [14, 15, 16], image question answering [17, 18], knowledge-grounded image question answering [19, 20, 21], image question answering with reading…”
Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (2024, neutral): “…MMInstruct (en & zh) [155] VQAv2 (en) [74], GQA (en) [98], OKVQA (en) [178], Visual7W (en) [317], MMInstruct (en & zh) [155], General QA VSR (en) [147], FSC147 (en) [197], Objects365-YorN (en) [208], Hateful-Memes (en) [111] GeoQA+ (en) [20], CLEVR-Math (en) [144], Super-CLEVR (en) [141], MapQA…”
What matters when building vision-language models? (2024, neutral): “…et al., 2023) use interleaved Transformer-based cross-attentions to fuse the image information into the language model, while (Li et al., 2023; Liu et al., 2023) concatenate the sequence of image hidden states with the sequence of text embeddings, and feed the concatenated sequence to the…”
LVLM-EHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models (2023, neutral): “Conclusion: This paper proposes a comprehensive evaluation benchmark for large vision-language models called LVLM-eHub that incorporates both quantitative performance evaluation and human feedback evaluation.”
MiniGPT-v2: large language model as a unified interface for vision-language multi-task learning (2023, methodology): “For VQA benchmarks, we consider OKVQA [43], GQA [19], visual spatial reasoning (VSR) [25], IconVQA [28], VizWiz [15], HatefulMemes and (HM) [21].”
Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models (2024, methodology): “…four tasks spanning object localization (Kazemzadeh et al., 2014; Yu et al., 2016; Wang et al., 2021), and four challenge tasks evaluating fine-grained spatial reasoning, hallucination, and diagram understanding (Acharya et al., 2018; Liu et al., 2022; Li et al., 2023d; Kembhavi et al., 2016).”
SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models (2023, methodology): “Furthermore, we evaluate general VQA benchmarks, such as VQAV2 (Agrawal et al., 2015), OKVQA (Marino et al., 2019), GQA (Hudson & Manning, 2019), vizwiz (Gurari et al., 2018), ScienceQA (Lu et al., 2022), visual spatial reasoning (VSR) (Liu et al., 2023a), IconQA (Lu et al., 2021).”
How far are we to GPT-4V? Closing the gap to commercial multimodal models with open-source suites (2024, neutral): “…stage. task dataset Captioning TextCaps (en) [95], ShareGPT4V (en&zh) [18] VQAv2 (en) [28], GQA (en) [34], OKVQA (en) [76], General QA VSR (en) [55], VisualDialog (en) [24] Science AI2D (en) [39], ScienceQA (en) [69], TQA (en) [40] ChartQA (en) [77], MMC-Inst (en) [57], DVQA (en) [38], Chart…”
SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities (2024, background): “Others have focused specifically on spatial reasoning as a task, answering questions about object spatial relations (e.g., above, below, left, right) in real [42, 52] or simulated [33] scenes.”
Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning (2024, methodology): “We select the Visual Spatial Reasoning (VSR) [32], GQA [18], and Open Images [23] datasets to construct data focusing on relation-reasoning.”
BLINK: Multimodal Large Language Models Can See but Not Perceive (2024, methodology): “To appropriately position our paper, let us revisit how we currently evaluate perception through using multimodal LLMs [43,44,47,52,53,57,87].”
MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuning (2022, methodology): “To build M ULTI I NSTRUCT , we first collect 34 tasks from the existing studies in visual and multimodal learning, covering Visual Question Answering (Goyal et al., 2017; Krishna et al., 2017; Zhu et al., 2016; Hudson and Manning, 2019; Singh et al., 2019; Marino et al., 2019), Commonsense Reasoning”
MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning (2023, methodology): “Visual Reasoning includes the Winoground (Thrush et al., 2022b), VSR (Liu et al., 2022) and IconQA (Lu et al., 2021) dataset.”