Driving Into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving

By Yu-Quan Wang, Jiawei He, Lue Fan, Hongxin Li, Yuntao Chen, Zhaoxiang Zhang. Computer Vision and Pattern Recognition, 2023

358 citations recorded by the source corpus.

In autonomous driving, predicting future events in advance and evaluating the foreseeable risks empowers autonomous vehicles to better plan their actions, enhancing safety and efficiency on the road. To this end, we propose Drive-Wm, the first driving world model compatible with existing end-to-end planning models. Through a joint spatial-temporal modeling facilitated by view factorization, our model generates high-fidelity multiview videos in driving scenes. Building on its powerful generation ability, we showcase the potential of applying the world model for safe driving planning for the first time. Particularly, our Drive-Wm enables driving into multiple futures based on distinct driving maneuvers, and determines the optimal trajectory according to the image-based rewards. Evaluation on real-world driving datasets verifies that our method could generate high-quality, consistent, and controllable multiview videos, opening up possibilities for real-world simulations and safe planning.

Subfields: Computer Vision and Pattern Recognition, Geography, Planning and Development, Signal Processing

View Paper: https://www.semanticscholar.org/paper/db2aae1bd6e71e7c23716d0eb50d12aa72a0f0f2

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Xinggang Wang
2. Hongyang Li
3. Qian Zhang
4. Kashyap Chitta
5. Andreas Geiger
6. Li Chen
7. Peng Wu
8. Bernhard Jaeger

In Signal Processing

1. Qian Zhang
2. Xinggang Wang
3. Kashyap Chitta
4. Hongyang Li
5. Andreas Geiger

Citations in the corpus

• End-to-End Autonomous Driving: Challenges and Frontiers (2023, methodology): “To tackle this, GenAD [203] and DriveWM [204] employ the prevailing diffusion technique [205].”
• Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability (2024, neutral): “Although a primary prospect of world models is to enable the generalization ability to novel environments, existing driving world models are still constrained by data scale [90, 125, 127, 143, 147] and geographical coverage [54, 61].”
• DiffusionDrive: Truncated Diffusion Model for End-to-End Autonomous Driving (2024, positive): “Subsequently, a series of works [6, 8, 12, 23, 26, 43, 45, 58] have adopted the single-trajectory planning paradigm to enhance planning performance further.”