DriveGPT4: Interpretable End-to-End Autonomous Driving Via Large Language Model

By Zhenhua Xu, Yujia Zhang, Enze Xie, Zhen Zhao, Yong Guo, Kwan-Yee Kenneth Wong, Zhenguo Li, Hengshuang Zhao. IEEE Robotics and Automation Letters, 2023

704 citations recorded by the source corpus.

Multimodallarge language models (MLLMs) have emerged as a prominent area of interest within the research community, given their proficiency in handling and reasoning with non-textual data, including images and videos. This study seeks to extend the application of MLLMs to the realm of autonomous driving by introducing DriveGPT4, a novel interpretable end-to-end autonomous driving system based on LLMs. Capable of processing multi-frame video inputs and textual queries, DriveGPT4 facilitates the interpretation of vehicle actions, offers pertinent reasoning, and effectively addresses a diverse range of questions posed by users. Furthermore, DriveGPT4 predicts low-level vehicle control signals in an end-to-end fashion. These advanced capabilities are achieved through the utilization of a bespoke visual instruction tuning dataset, specifically tailored for autonomous driving applications, in conjunction with a mix-finetuning training strategy. DriveGPT4 represents the pioneering effort to leverage LLMs for the development of an interpretable end-to-end autonomous driving solution. Evaluations conducted on the BDD-X dataset showcase the superior qualitative and quantitative performance of DriveGPT4. Additionally, the fine-tuning of domain-specific data enables DriveGPT4 to yield close or even improved results in terms of autonomous driving grounding when contrasted with GPT4-V.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/ccd6f8b6544f112de632e49bfbe592a0a654537d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

VADv2: End-to-End Vectorized Autonomous Driving via Probabilistic Planning (2024, background): “Recent research has explored the combination of LLMs and autonomous driving[7, 10, 12, 44, 51, 34, 31, 50, 49].”
End-to-End Autonomous Driving: Challenges and Frontiers (2023, background): “Recently, observing the rapid development of large language models (LLMs) [166], [167], works encode the perceived scene into tokens and prompt them to LLMs for control prediction and text-based explanations [168], [169], [170].”
DriveLM: Driving with Graph Visual Question Answering (2023, methodology): “DriveGPT4 [81] projects raw sensor data into tokens and utilizes LLMs for end-to-end prediction of con-trol signals and explanations.”
VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs (2024, background): “…2023; Song et al., 2023; Wang et al., 2024c), video agent (Lin et al., 2023b; Fan et al., 2024; Wang et al., 2024a; He et al., 2024), autonomous driving (Xu et al., 2023; Shao et al., 2024), motion understanding (Wu et al., 2024; Chen et al., 2024a), and robotic manipulation (Liu et al., 2024d).”
LMDrive: Closed-Loop End-to-End Driving with Large Language Models (2023, background): “For example, DRIVEGPT4 [52] proposed a multimodal LLM framework, which takes a sequence of frames as input, then generates responses to human inquiries and predicts control signals for the next step.”