GPT-Driver: Learning to Drive with GPT

By Jiageng Mao, Yuxi Qian, Hang Zhao, Yue Wang. arXiv.org, 2023

449 citations recorded by the source corpus.

We present a simple yet effective approach that can transform the OpenAI GPT-3.5 model into a reliable motion planner for autonomous vehicles. Motion planning is a core challenge in autonomous driving, aiming to plan a driving trajectory that is safe and comfortable. Existing motion planners predominantly leverage heuristic methods to forecast driving trajectories, yet these approaches demonstrate insufficient generalization capabilities in the face of novel and unseen driving scenarios. In this paper, we propose a novel approach to motion planning that capitalizes on the strong reasoning capabilities and generalization potential inherent to Large Language Models (LLMs). The fundamental insight of our approach is the reformulation of motion planning as a language modeling problem, a perspective not previously explored. Specifically, we represent the planner inputs and outputs as language tokens, and leverage the LLM to generate driving trajectories through a language description of coordinate positions. Furthermore, we propose a novel prompting-reasoning-finetuning strategy to stimulate the numerical reasoning potential of the LLM. With this strategy, the LLM can describe highly precise trajectory coordinates and also its internal decision-making process in natural language. We evaluate our approach on the large-scale nuScenes dataset, and extensive experiments substantiate the effectiveness, generalization ability, and interpretability of our GPT-based motion planner. Code is now available at https://github.com/PointsCoder/GPT-Driver.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/958ed4830ae80a189ecb9b93ab75a6ce2e3926fc

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

DriveLM: Driving with Graph Visual Question Answering (2023, methodology): “Specifically, GPT-Driver [48] and LLM-Driver [7] encode the perceived scene state into prompts, relying on LLMs to formulate reasonable plans.”
End-to-End Autonomous Driving: Challenges and Frontiers (2023, background): “Recently, observing the rapid development of large language models (LLMs) [166], [167], works encode the perceived scene into tokens and prompt them to LLMs for control prediction and text-based explanations [168], [169], [170].”
VADv2: End-to-End Vectorized Autonomous Driving via Probabilistic Planning (2024, background): “Recent research has explored the combination of LLMs and autonomous driving[7, 10, 12, 44, 51, 34, 31, 50, 49].”
LMDrive: Closed-Loop End-to-End Driving with Large Language Models (2023, methodology): “…textual descriptions as the input for the LLM, for directly interpreting and reasoning about comprehensive driving situations In this thread of works, GPT-Driver [32] reformulated motion planning as the task of natural language modeling by converting heterogeneous scene input to language tokens.”
A Survey on Multimodal Large Language Models for Autonomous Driving (2023, background): “Moreover, the Waymo Open Sim Agents Challenge (WOSAC) [50,117] is the first public challenge to develop simulations with realistic and interactive agents.”