SQA3D: Situated Question Answering in 3D Scenes

By Xiaojian Ma, Silong Yong, Zilong Zheng, Qing Li, Yitao Liang, Song-Chun Zhu, Siyuan Huang. International Conference on Learning Representations, 2022

348 citations recorded by the source corpus.

We propose a new task to benchmark scene understanding of embodied agents: Situated Question Answering in 3D Scenes (SQA3D). Given a scene context (e.g., 3D scan), SQA3D requires the tested agent to first understand its situation (position, orientation, etc.) in the 3D scene as described by text, then reason about its surrounding environment and answer a question under that situation. Based upon 650 scenes from ScanNet, we provide a dataset centered around 6.8k unique situations, along with 20.4k descriptions and 33.4k diverse reasoning questions for these situations. These questions examine a wide spectrum of reasoning capabilities for an intelligent agent, ranging from spatial relation comprehension to commonsense understanding, navigation, and multi-hop reasoning. SQA3D imposes a significant challenge to current multi-modal especially 3D reasoning models. We evaluate various state-of-the-art approaches and find that the best one only achieves an overall score of 47.20%, while amateur human participants can reach 90.06%. We believe SQA3D could facilitate future embodied AI research with stronger situation understanding and reasoning capability.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/c1a4fb211cf995b1af1247a152fcc145594ec13b

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenario (2023, background): “In contrast, other datasets [3, 8, 31, 39, 41, 46] typically have less than 1000 scenes due to the difficulty of acquiring 3D data.”
OpenEQA: Embodied Question Answering in the Era of Foundation Models (2024, positive): “To ensure coverage of the answer distributions, we sampled responses from blind LLaMA-2, GPT-4V, and human annotated answers.”
Aligning Cyber Space With Physical World: A Comprehensive Survey on Embodied AI (2024, neutral): “SQA3D [141] sim-pliﬁes protocol (QA only) while still preserving the function of benchmarking embodied scene understanding, enabling more complex, knowledge-intensivequestions andamuchlarger scale of data collection.”
Diffusion-based Generation, Optimization, and Planning in 3D Scenes (2023, background): “Planning in 3D Scenes The ability to act and plan in 3D scenes is critical for an intelligent agent and has led to the recent culmination of embodied AI research [28, 30, 33, 36, 54, 55, 79].”
Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoning (2022, background): “Large pre-trained language models, such as GPT-3 (Brown et al., 2020), have shown their remarkable ability of few-shot learning on a wide range of downstream tasks (Houlsby et al., 2019; Brown et al., 2020; Ma et al., 2022; Lu et al., 2022a).”
Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agents (2023, background): “Developing multi-task agents that can accomplish a vast and diverse suite of tasks in complex domains has been viewed as one of the key milestones towards generally capable artificial intelligence [34, 1, 5, 10, 25].”
An Embodied Generalist Agent in 3D World (2023, neutral): “Where is the bathtub located in the bathroom?”
eMEM: A Hybrid Spatio-Temporal Memory System For Embodied Agents (2026, unknown): “SQA3D [Ma et al., 2023] tests situated spatial reasoning from a fixed pose, not memory recall over time.”