MP3: A Unified Model to Map, Perceive, Predict and Plan

By Sergio Casas, Abbas Sadat, Raquel Urtasun. Computer Vision and Pattern Recognition, 2021

301 citations recorded by the source corpus.

High-definition maps (HD maps) are a key component of most modern self-driving systems due to their valuable semantic and geometric information. Unfortunately, building HD maps has proven hard to scale due to their cost as well as the requirements they impose in the localization system that has to work everywhere with centimeter-level accuracy. Being able to drive without an HD map would be very beneficial to scale self-driving solutions as well as to increase the failure tolerance of existing ones (e.g., if localization fails or the map is not up-to-date). Towards this goal, we propose MP3, an end-to-end approach to mapless1 driving where the input is raw sensor data and a high-level command (e.g., turn left at the intersection). MP3 predicts intermediate representations in the form of an online map and the current and future state of dynamic agents, and exploits them in a novel neural motion planner to make interpretable decisions taking into account uncertainty. We show that our approach is significantly safer, more comfortable, and can follow commands better than the baselines in challenging long-term closed-loop simulations, as well as when compared to an expert driver in a large-scale real-world dataset.

Subfields: Automotive Engineering, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/1ba56df34200b028eace2f6cbdd62cb8bc2de20c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Automotive Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

End-to-End Autonomous Driving: Challenges and Frontiers (2023, methodology): “Recent works prioritize generating safety-critical data [7], [34], [35], pre-training a foundation model or backbone curated for policy learning [12], [36], [37], and advocating a modular end-to-end planning philosophy [1], [2], [38], [39].”
GPT-Driver: Learning to Drive with GPT (2023, methodology): “…learning-based approaches Bojarski et al. (2016); Codevilla et al. (2018; 2019); Rhinehart et al. (2019); Zeng et al. (2019); Sadat et al. (2020); Casas et al. (2021); Hu et al. (2022; 2023); Dauner et al. (2023) resorted to a data-driven strategy and learned their models from large-scale human…”
MultiPath++: Efficient Information Fusion and Trajectory Aggregation for Behavior Prediction (2021, background): “Notably unique are (1) HOME [22] and GOHOME [21] which first predict a heatmap, and then decode trajectories after sampling, and (2) MP3 [11] and NMP [52] which learn a cost function evaluator of trajectories, and the trajectories are enumerated heuristically rather than generated by a learned model”
Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baseline (2022, background): “[53] train a cost volume to generate the planning route, while [45, 8] explicitly design safety and comfort costs based on semantic occupancy maps to select the best one in the expert trajectory sets.”
Parting with Misconceptions about Learning-based Vehicle Motion Planning (2023, background): “This forecasting can either be agent-centric [30, 31, 32], where trajectories are determined for each actor, or environment-centric [4, 33, 34, 20, 35, 36], involving occupancy or cost maps.”
VAD: Vectorized Scene Representation for Efficient Autonomous Driving (2023, methodology): “3.4).”
3D Object Detection for Autonomous Driving: A Comprehensive Survey (2022, background): “[246] introduces a semantic occupancy map to produce interpretable intermediate representations, [310] incorporates spatial attention into the framework, [363] proposes a deep structured network, [23] proposes a map-free approach, [53] produces a diverse set of future trajectories.”
DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models (2024, unknown): “Learning-based Planning.”
Driving Into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving (2023, methodology): “Then we define map reward and object reward, inspired by traditional planner [6, 30].”
Recent Advancements in End-to-End Autonomous Driving Using Deep Learning: A Survey (2023, methodology): “The evaluation metrics commonly used in the open-loop evaluation include measures of the system’s ability to follow the desired trajectory or driving behaviors, such as the mean squared error [50], L2 [9, 82] between the predicted and actual trajectories or the percentage of time the system remains ”
Motion Transformer with Global Intention Localization and Local Movement Refinement (2022, background): “To encode such scene context, early works [36, 31, 5, 12, 60, 3, 8] typically rasterize them into an image so as to be processed with convolutional neural networks (CNNs).”
ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learning (2022, methodology): “In this paper, concurrent to the interpretable LiDAR-based pipelines [5, 52], we propose to investigate the potential of vision-based end-to-end framework (Fig.”
NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarking (2024, unknown): “While the idea of sampling and scoring trajectories is well-known [46, 51, 55, 6, 16], it has recently been overlooked in favor of approaches which predict a single trajectory.”
VectorMapNet: End-to-end Vectorized HD Map Learning (2022, methodology): “Some methods (Casas et al., 2021; Sadat et al., 2020) use pixel-level semantic maps to solve down-stream tasks, but the entire downstream pipeline needs to be redesigned to accommodate these rasterized map inputs.”
Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability (2024, unknown): “Moreover, most of them are restricted to a specific control modality [54, 61, 90, 125], which hinders the accommodation to prevailing planning algorithms [12, 14, 21, 56, 58, 64] and extension to more applications like decision-making [127] or user interaction [74].”
nuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles (2021, background): “MP3 [5] proposes an end-to-end approach to mapless driving, where the input is raw lidar data and a high-level navigation goal.”
Planning-oriented Autonomous Driving (2022, background): “By contrast, the emergence of end-to-end autonomous driving [11,15,19,38,97] unites all nodes from perception, prediction and planning as a whole .”
Diffusion-based Generation, Optimization, and Planning in 3D Scenes (2023, background): “igation [1, 88], embodied perception and manipulation [30, 37, 59], and autonomous driving [3, 50].”
VADv2: End-to-End Vectorized Autonomous Driving via Probabilistic Planning (2024, background): “Imitation learning [2, 18, 19, 23] is another research direction, where models learn expert driving behavior to achieve good planning performance and develop a driving style close to that of humans.”