DriveLM: Driving with Graph Visual Question Answering

By Chonghao Sima, Katrin Renz, Kashyap Chitta, Li Chen, Hanxue Zhang, Chengen Xie, Ping Luo, Andreas Geiger, Hongyang Li. European Conference on Computer Vision, 2023

601 citations recorded by the source corpus.

We study how vision-language models (VLMs) trained on web-scale data can be integrated into end-to-end driving systems to boost generalization and enable interactivity with human users. While recent approaches adapt VLMs to driving via single-round visual question answering (VQA), human drivers reason about decisions in multiple steps. Starting from the localization of key objects, humans estimate object interactions before taking actions. The key insight is that with our proposed task, Graph VQA, where we model graph-structured reasoning through perception, prediction and planning question-answer pairs, we obtain a suitable proxy task to mimic the human reasoning process. We instantiate datasets (DriveLM-Data) built upon nuScenes and CARLA, and propose a VLM-based baseline approach (DriveLM-Agent) for jointly performing Graph VQA and end-to-end driving. The experiments demonstrate that Graph VQA provides a simple, principled framework for reasoning about a driving scene, and DriveLM-Data provides a challenging benchmark for this task. Our DriveLM-Agent baseline performs end-to-end autonomous driving competitively in comparison to state-of-the-art driving-specific architectures. Notably, its benefits are pronounced when it is evaluated zero-shot on unseen objects or sensor configurations. We hope this work can be the starting point to shed new light on how to apply VLMs for autonomous driving. To facilitate future research, all code, data, and models are available to the public.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/3c8cc9a5ee373d51e0bf71621b6eb6901c762e8f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Zhen Zhao
2. Hengshuang Zhao
3. Enze Xie
4. Zhenguo Li

In Computer Vision and Pattern Recognition

1. Zhenguo Li
2. Enze Xie
3. Hengshuang Zhao
4. Zhen Zhao
5. Yong Guo
6. Yujia Zhang
7. Zhenhua Xu
8. Kwan-Yee Kenneth Wong

Citations in the corpus

• VADv2: End-to-End Vectorized Autonomous Driving via Probabilistic Planning (2024, neutral): “Closed-loop demos show it runs stably in an end-to-end manner.”
• DriveGPT4: Interpretable End-to-End Autonomous Driving Via Large Language Model (2023, background): “Among these works, only a few focus on autonomous driving-related tasks [44], [45], [46].”
• NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarking (2024, neutral): “The top methods ranged from multi-billion parameter vision language models [45, 33, 53, 58] to more efficient and recently overlooked approaches based on trajectory sampling and scoring [40, 20, 10], demonstrating the remarkable ability of the broader community to advance AV research when provided…”
• Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability (2024, neutral): “Since nuScenes is heavily biased [83, 107, 127], we balance its samples based on command categories to foster the learning of rare actions.”
• Why Braking? Scenario Extraction and Reasoning Utilizing LLM (2025, unknown): “Recent works such as DriveLM [12] and DriveVLM [13] integrate LLM into end-to-end autonomous driving systems, focusing primarily on single-scene understanding based on visual inputs.”