End-to-End Autonomous Driving: Challenges and Frontiers

By Li Chen, Peng Wu, Kashyap Chitta, Bernhard Jaeger, Andreas Geiger, Hongyang Li. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023

874 citations recorded by the source corpus.

The autonomous driving community has witnessed a rapid growth in approaches that embrace an end-to-end algorithm framework, utilizing raw sensor input to generate vehicle motion plans, instead of concentrating on individual tasks such as detection and motion prediction. End-to-end systems, in comparison to modular pipelines, benefit from joint feature optimization for perception and planning. This field has flourished due to the availability of large-scale datasets, closed-loop evaluation, and the increasing need for autonomous driving algorithms to perform effectively in challenging scenarios. In this survey, we provide a comprehensive analysis of more than 270 papers, covering the motivation, roadmap, methodology, challenges, and future trends in end-to-end autonomous driving. We delve into several critical challenges, including multi-modality, interpretability, causal confusion, robustness, and world models, amongst others. Additionally, we discuss current advancements in foundation models and visual pre-training, as well as how to incorporate these techniques within the end-to-end driving framework.

Subfields: Automotive Engineering

View Paper: https://www.semanticscholar.org/paper/318128fa82a15888a5db28341c5c23d1147271f3

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Automotive Engineering

Citations in the corpus

DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models (2024, background): “One promising line of work is Reinforcement learning and imitation learning [8, 9, 43].”
LMDrive: Closed-Loop End-to-End Driving with Large Language Models (2023, background): “Much progress [7, 8] has been achieved recently in the field of end-to-end autonomous driving.”
NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarking (2024, neutral): “Autonomous vehicles (AVs) have gained immense research interest due to their potential to change transportation and improve traffic safety [23, 9].”
Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability (2024, neutral): “It also holds the promise to remedy the irrationality in current evaluation protocols for planning [18, 83, 141], such as the L2 error shown in Fig.”
DriveLM: Driving with Graph Visual Question Answering (2023, methodology): “Inspired by [8], DriveLM considers Graph Visual Question Answering (GVQA) , where question-answer pairs are interconnected via logical dependencies at the object-level, i.e. , interactions between object pairs, and the task-level, e.g. , perception → prediction → planning → behavior (discretized…”
Recent Advancements in End-to-End Autonomous Driving Using Deep Learning: A Survey (2023, methodology): “[29] discuss the methodology and challenges in end-to-end autonomous driving in their survey.”
DriveGPT4: Interpretable End-to-End Autonomous Driving Via Large Language Model (2023, background): “To better integrate and optimize the entire system, some approaches pro-pose training the network in an end-to-end manner, eliminating the need for discontinuous intermediate steps [8], [9], [10].”
Parting with Misconceptions about Learning-based Vehicle Motion Planning (2023, methodology): “Ego-forecasting approaches include both end-to-end methods [43] that utilize LiDAR scans [44, 45], RGB images [46, 47, 48, 49, 14, 50] or both [13, 51, 5, 52], as well as modular methods involving lower-dimensional inputs like bird’s eye view (BEV) grids or state vectors [24, 53, 11, 54, 55, 56].”
A Training-Free Approach to Long-Tail Object Detection for Autonomous Driving (2026, unknown): “This makes it possible to build systems that can automatically label challenging, complex, and long-tailed datasets, laying the foundation for advancing data-driven autonomous driving [2].”