LMDrive: Closed-Loop End-to-End Driving with Large Language Models

By Hao Shao, Yuxuan Hu, Letian Wang, Steven L. Waslander, Yu Liu, Hongsheng Li. Computer Vision and Pattern Recognition, 2023

378 citations recorded by the source corpus.

Despite significant recent progress in the field of autonomous driving, modern methods still struggle and can incur serious accidents when encountering long-tail unfore-seen events and challenging urban scenarios. On the one hand, large language models (LLM) have shown impres-sive reasoning capabilities that approach “Artificial Gen-eral Intelligence”. On the other hand, previous autonomous driving methods tend to rely on limited-format inputs (e.g., sensor data and navigation waypoints), restricting the vehi-cle's ability to understand language information and inter-act with humans. To this end, this paper introduces LM-Drive, a novel language-guided, end-to-end, closed-loop autonomous driving framework. LMDrive uniquely processes and integrates multimodal sensor data with naturallanguage instructions, enabling interaction with humans and navigation software in realistic instructional settings. To facilitate research in language-based closed-loop autonomous driving, we also publicly release the corresponding dataset which includes approximately 64K instruction-following data clips, and the LangAuto benchmark that tests the system's ability to handle complex instructions and challenging driving scenarios. Extensive closed-loop experiments are conducted to demonstrate LMDrive's effectiveness. To the best of our knowledge, we're the very first work to leverage LLMs for closed-loop end-to-end autonomous driving. Code is available on our webpage.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/e0b05e314372ed580d9612ef5f0ee672b17ad2e4

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Peng Wu
2. Jiwen Yu
3. Bernhard Jaeger
4. Li Chen
5. Wanli Ouyang
6. Lei Bai
7. Kashyap Chitta
8. Hongyang Li

In Computer Vision and Pattern Recognition

1. Hongyang Li
2. Kashyap Chitta
3. Li Chen
4. Peng Wu
5. Bernhard Jaeger
6. Andreas Geiger
7. Jiwen Yu
8. Wanli Ouyang

Citations in the corpus

• DriveLM: Driving with Graph Visual Question Answering (2023, neutral): “Finally, we introduce DriveLM-CARLA as a means to show closed-loop planning results in the future.”
• WorldSimBench: Towards Video Generation Models as World Simulators (2024, methodology): “For Video Generation Model training, we use LMDrive Training Dataset (Shao et al., 2024).”
• End-to-End Autonomous Driving: Challenges and Frontiers (2023, background): “Recently, observing the rapid development of large language models (LLMs) [166], [167], works encode the perceived scene into tokens and prompt them to LLMs for control prediction and text-based explanations [168], [169], [170].”
• Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning (2024, neutral): “However, many popular MLLMs are primarily trained to respond to instructions based on visual inputs, employing a decoder-only autoregressive design as a single black box.”