Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Driving

By Long Chen, Oleg Sinavski, Jan Hünermann, Alice Karnsund, Andrew J Willmott, Danny Birch, Daniel Maund, Jamie Shotton. IEEE International Conference on Robotics and Automation, 2023

357 citations recorded by the source corpus.

Large Language Models (LLMs) have shown promise in the autonomous driving sector, particularly in generalization and interpretability. We introduce a unique objectlevel multimodal LLM architecture that merges vectorized numeric modalities with a pre-trained LLM to improve context understanding in driving situations. We also present a new dataset of 160k QA pairs derived from 10k driving scenarios, paired with high quality control commands collected with RL agent and question answer pairs generated by teacher LLM (GPT-3.5). A distinct pretraining strategy is devised to align numeric vector modalities with static LLM representations using vector captioning language data. We also introduce an evaluation metric for Driving QA and demonstrate our LLM-driver’s proficiency in interpreting driving scenarios, answering questions, and decision-making. Our findings highlight the potential of LLM-based driving action generation in comparison to traditional behavioral cloning. We make our benchmark, datasets, and model available1 for further exploration.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/f01ff5acf9e086030c01beda6f433f99013ebbd4

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

VADv2: End-to-End Vectorized Autonomous Driving via Probabilistic Planning (2024, background): “Recent research has explored the combination of LLMs and autonomous driving[7, 10, 12, 44, 51, 34, 31, 50, 49].”
Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding (2023, unknown): “In domains like healthcare [26, 66], autonomous systems [8, 69], and robotics [46, 50], such inaccuracies are not just undesirable but could have significant consequences.”
LMDrive: Closed-Loop End-to-End Driving with Large Language Models (2023, methodology): “One of the most common strategies [6, 11, 32, 36] is to 1) first use LLMs to transform the scene perception re-sults and navigation commands into textual descriptions; 2) feed these textural descriptions into LLMs to generate textual driving decisions; and then 3) transfer textual driving decisions…”
A Survey on Multimodal Large Language Models for Autonomous Driving (2023, background): “Moreover, the Waymo Open Sim Agents Challenge (WOSAC) [50,117] is the first public challenge to develop simulations with realistic and interactive agents.”
DriveLM: Driving with Graph Visual Question Answering (2023, methodology): “Specifically, GPT-Driver [48] and LLM-Driver [7] encode the perceived scene state into prompts, relying on LLMs to formulate reasonable plans.”