End-To-End Interpretable Neural Motion Planner

By Wenyuan Zeng, Wenjie Luo, Simon Suo, Abbas Sadat, Binh Yang, Sergio Casas, Raquel Urtasun. Computer Vision and Pattern Recognition, 2019

482 citations recorded by the source corpus.

In this paper, we propose a neural motion planner for learning to drive autonomously in complex urban scenarios that include traffic-light handling, yielding, and interactions with multiple road-users. Towards this goal, we design a holistic model that takes as input raw LIDAR data and a HD map and produces interpretable intermediate representations in the form of 3D detections and their future trajectories, as well as a cost volume defining the goodness of each position that the self-driving car can take within the planning horizon. We then sample a set of diverse physically possible trajectories and choose the one with the minimum learned cost. Importantly, our cost volume is able to naturally capture multi-modality. We demonstrate the effectiveness of our approach in real-world driving data captured in several cities in North America. Our experiments show that the learned cost volume can generate safer planning than all the baselines.

Subfields: Automotive Engineering, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/1a8817eea8388580124eee61544fba183904fb29

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Automotive Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Semantic Segmentation with Generative Models: Semi-Supervised Learning and Strong Out-of-Domain Generalization (2021, background): “Deep learning is now powering the majority of computer vision applications ranging from autonomous driving [93, 73] and medical imaging [78, 38] to image editing [69, 15, 98, 88, 70, 74].”
End-to-End Urban Driving by Imitating a Reinforcement Learning Coach (2021, background): “Numerous works have studied different network architectures for this task [3, 16, 52], yet most of these approaches use supervised learning with expert demonstrations, which is known to suffer from covariate shift [36, 40].”
Learning Lane Graph Representations for Motion Forecasting (2020, methodology): “Rasterized maps have been fused with LiDAR point clouds to perform joint perception and prediction [29, 4, 27] as well as end-to-end motion planning [40, 35, 41].”
Combining Optimal Control and Learning for Visual Navigation in Novel Environments (2019, background): “Several works also use end-to-end learning for navigation using laser scans [12, 13, 14], for training and combining a local planner with a higher level roadmap for long range navigation [15, 16, 17, 18], or visual servoing [19].”
Trajectron++: Dynamically-Feasible Trajectory Forecasting with Heterogeneous Data (2020, background): “hat do incorporate additional data produce trajectory forecasts from raw point clouds, high-deﬁnition semantic maps, and their histories, all showing performance improvements over previous approaches [44,8,20,7]. These approaches generally make use of end-to-end learning architectures that encode ra”
MP3: A Unified Model to Map, Perceive, Predict and Plan (2021, methodology): “Neural Motion Plan-ner (NMP) [55], where a planning cost-volume as well as detection and prediction are predicted in a multi-task fashion from the scene context features, and Trajectory Classiﬁca-tion (TC) [37], where a cost-volume is predicted similar to NMP, but the trajectory cost is used to…”
Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D (2020, methodology): “o the template trajectories Tthen train with the cross entropy loss. This denition of p(˝ ijo) enables us to learn an interpretable spatial cost function without dening a hard-margin loss as in NMP [41]. In practice, we determine the set of template trajectories by running K-Means on a large number ”
GPT-Driver: Learning to Drive with GPT (2023, methodology): “Alternatively, the learning-based approaches Bojarski et al. (2016); Codevilla et al. (2018; 2019); Rhinehart et al. (2019); Zeng et al. (2019); Sadat et al. (2020); Casas et al. (2021); Hu et al. (2022; 2023); Dauner et al. (2023) resorted to a data-driven strategy and learned their models from…”
DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models (2024, background): “Specifically, given instructions, input images, and robot states, vision language models output corresponding actions that can be high-level instructions [15] or low-level robot actions [5].”
Recent Advancements in End-to-End Autonomous Driving Using Deep Learning: A Survey (2023, methodology): “[84] employ a neural motion planner that uses a cost volume to predict future trajectories.”
NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarking (2024, unknown): “While the idea of sampling and scoring trajectories is well-known [46, 51, 55, 6, 16], it has recently been overlooked in favor of approaches which predict a single trajectory.”
StarNet: Targeted Computation for Object Detection in Point Clouds (2019, background): “Additionally, in SDC perception, every location in the scene is not equally important [50, 4, 2], and that importance can change dynamically based on the local environment and context.”
ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learning (2022, methodology): “In this paper, concurrent to the interpretable LiDAR-based pipelines [5, 52], we propose to investigate the potential of vision-based end-to-end framework (Fig.”
Planning-oriented Autonomous Driving (2022, background): “This is a popular practice in many domains, including general vision [46, 51, 61], autonomous driving2 [8, 34, 57, 59], such as Transfuser [13], BEV-”