NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenario

By Tianwen Qian, Jingjing Chen, Linhai Zhuo, Yang Jiao, Yu-Gang Jiang. AAAI Conference on Artificial Intelligence, 2023

355 citations recorded by the source corpus.

We introduce a novel visual question answering (VQA) task in the context of autonomous driving, aiming to answer natural language questions based on street-view clues. Compared to traditional VQA tasks, VQA in autonomous driving scenario presents more challenges. Firstly, the raw visual data are multi-modal, including images and point clouds captured by camera and LiDAR, respectively. Secondly, the data are multi-frame due to the continuous, real-time acquisition. Thirdly, the outdoor scenes exhibit both moving foreground and static background. Existing VQA benchmarks fail to adequately address these complexities. To bridge this gap, we propose NuScenes-QA, the first benchmark for VQA in the autonomous driving scenario, encompassing 34K visual scenes and 460K question-answer pairs. Specifically, we leverage existing 3D detection annotations to generate scene graphs and design question templates manually. Subsequently, the question-answer pairs are generated programmatically based on these templates. Comprehensive statistics prove that our NuScenes-QA is a balanced large-scale benchmark with diverse question formats. Built upon it, we develop a series of baselines that employ advanced 3D detection and VQA techniques. Our extensive experiments highlight the challenges posed by this new task. Codes and dataset are available at https://github.com/qiantianwen/NuScenes-QA.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/ee156428803c5bd6e7372f6b27d74bcf88390db3

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

A Survey on Multimodal Large Language Models for Autonomous Driving (2023, methodology): “These datasets set new standards by offering a large number of scenes.”
DriveLM: Driving with Graph Visual Question Answering (2023, neutral): “[47, 55, 59].”
End-to-End Autonomous Driving: Challenges and Frontiers (2023, background): “Recently, [169], [171], [172] resort to the progress of multi-modality and foundation models, using LLMs/VLMs to provide decision-related explanations, as discussed in Section IV-A-2.”
DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models (2024, background): “However, the direct generation of control outputs from sensor data poses significant challenges in robustness and safety assurance [11].”