ScanQA: 3D Question Answering for Spatial Scene Understanding

By Daich Azuma, Taiki Miyanishi, Shuhei Kurita, Motoaki Kawanabe. Computer Vision and Pattern Recognition, 2021

447 citations recorded by the source corpus.

We propose a new 3D spatial understanding task for 3D question answering (3D-QA). In the 3D-QA task, models receive visual information from the entire 3D scene of a rich RGB-D indoor scan and answer given textual questions about the 3D scene. Unlike the 2D-question answering of visual question answering, the conventional 2D-QA models suffer from problems with spatial understanding of object alignment and directions and fail in object localization from the textual questions in 3D-QA. We propose a baseline model for 3D-QA, called the ScanQA11https://github.com/ATR-DBI/ScanQA, which learns a fused descriptor from 3D object proposals and encoded sentence embeddings. This learned descriptor correlates language expressions with the underlying geometric features of the 3D scan and facilitates the regression of 3D bounding boxes to determine the described objects in textual questions. We collected human-edited question-answer pairs with free-form answers grounded in 3D objects in each 3D scene. Our new ScanQA dataset contains over 41k question-answer pairs from 800 indoor scenes obtained from the ScanNet dataset. To the best of our knowledge, ScanQA is the first large-scale effort to perform object-grounded question answering in 3D environments.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/8f6c652a392995bd047a2f7b94474ab1e6e23ff0

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Following (2023, background): “To extend its application scenarios, numerous efforts [1,23, 92,95] have been made to incorporate 3D point clouds with data from other modalities, allowing for improved 3D understanding [1, 23], text-to-3D generation [35, 49, 52], and 3D question answering [3, 28].”
Decomposing NeRF for Editing via Feature Field Distillation (2022, methodology): “Furthermore, while this work focuses on editable view synthesis, it is also intriguing to transfer DFF to other applications, including 3D registration of text queries [16, 1, 47, 4] or robot teaching [32, 80].”
LERF: Language Embedded Radiance Fields (2023, background): “3D Language Grounding Incorporating language into 3D has been explored in a wide range of contexts: 3D visual question answering [15, 2, 6] leverage 3D information to extract answers to queries about the environment.”
An Embodied Generalist Agent in 3D World (2023, unknown): “The kitchen features a cabinet and a counter.”
OpenEQA: Embodied Question Answering in the Era of Foundation Models (2024, unknown): “Speciﬁcally, we study two variants of EQA under a common umbrella: episodic memory (EM-EQA) and active exploration (A-EQA), depending on the agent platform.”
LangSplat: 3D Language Gaussian Splatting (2023, background): “Modeling a 3D language field allows users to interact with and query 3D worlds using open-ended language, which presents a promising avenue for human-computer interaction and understanding [1, 5, 13].”
LLaVA-OneVision: Easy Visual Task Transfer (2024, unknown): “We also utilize several multi-view benchmarks for evaluation, which depict 3D environments with multiple viewpoints, including 3D Dialogue (3D-Chat) and Task Decomposition (3D-TD) from 3D-LLM [38], ScanQA [5], ALFRED [122], and nuScenes VQA [9].”
SQA3D: Situated Question Answering in 3D Scenes (2022, methodology): “In our experiments, we examine state-of-the-art multi-modal reasoning models, including ScanQA from Azuma et al. (2022) that leverages 3D scan data, Clip-2 In general, we consider semantic grounding, language-driven navigation, and question-answering in photo-realistic 3D scenes.”
3D-LLM: Injecting the 3D World into Large Language Models (2023, methodology): “Another line of research that is similar to ours is 3D and language [5, 45, 8, 20, 1, 16, 22, 45, 3].”
NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenario (2023, methodology): “Most 2D and 3D VQA models [3, 38] fuse the visual feature with object bounding box in the object embedding stage, making it position-aware.”
LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models (2024, methodology): “For the 3D data, we widely gather the training set from nuScenes QA [6], ALFRED [48], ScanQA [3], and 3D-LLM [16], covering both outdoor and indoor scenarios.”
Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (2024, unknown): “…(en) [307], FineVideo (en) [67], PerceptionTest (en) [193], HiREST (en) [291], STAR (en) [259], General QA EgoSchema (en) [175], ScanQA (en) [10], LSMDC (en) [201] GUI GUI-World (en) [24] Type: Text Datasets UltraFeedback (en) [48], UltraChat (en) [58], Unnatural-Instructions (en) [90],…”