DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models
By Xiaoyu Tian, Junru Gu, Bailin Li, Yicheng Liu, Chenxu Hu, Yang Wang, Kun Zhan, Peng Jia, Xianpeng Lang, Hang Zhao. Conference on Robot Learning, 2024
536 citations recorded by the source corpus.
A primary hurdle of autonomous driving in urban environments is understanding complex and long-tail scenarios, such as challenging road conditions and delicate human behaviors. We introduce DriveVLM, an autonomous driving system leveraging Vision-Language Models (VLMs) for enhanced scene understanding and planning capabilities. DriveVLM integrates a unique combination of reasoning modules for scene description, scene analysis, and hierarchical planning. Furthermore, recognizing the limitations of VLMs in spatial reasoning and heavy computational requirements, we propose DriveVLM-Dual, a hybrid system that synergizes the strengths of DriveVLM with the traditional autonomous driving pipeline. Experiments on both the nuScenes dataset and our SUP-AD dataset demonstrate the efficacy of DriveVLM and DriveVLM-Dual in handling complex and unpredictable driving conditions. Finally, we deploy the DriveVLM-Dual on a production vehicle, verifying it is effective in real-world autonomous driving environments.
Subfields: Computer Vision and Pattern Recognition
View Paper: https://www.semanticscholar.org/paper/758c2dc290c037a6f211ec503beee70abe2d1197
Citations in the corpus
- DriveLM: Driving with Graph Visual Question Answering (2023, unknown): “Finally, we introduce DriveLM-CARLA as a means to show closed-loop planning results in the future.”
- Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces (2024, unknown): “While Large Language Models (LLMs) [3, 6, 9, 36, 61, 67, 68, 77, 81, 82, 88, 105] have advanced linguistic intelligence, visual-spatial intelligence remains under-explored, despite its relevance to robotics [7, 8, 21, 64], autonomous driving [79], and AR/VR [12, 27, 55].”
- Why Braking? Scenario Extraction and Reasoning Utilizing LLM (2025, unknown): “Recent works such as DriveLM [12] and DriveVLM [13] integrate LLM into end-to-end autonomous driving systems, focusing primarily on single-scene understanding based on visual inputs.”