NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarking

By Daniel Dauner, Marcel Hallgarten, Tianyu Li, Xinshuo Weng, Zhiyu Huang, Zetong Yang, Hongyang Li, Igor Gilitschenski, Boris Ivanovic, Marco Pavone and 2 more. Neural Information Processing Systems, 2024

348 citations recorded by the source corpus.

Benchmarking vision-based driving policies is challenging. On one hand, open-loop evaluation with real data is easy, but these results do not reflect closed-loop performance. On the other, closed-loop evaluation is possible in simulation, but is hard to scale due to its significant computational demands. Further, the simulators available today exhibit a large domain gap to real data. This has resulted in an inability to draw clear conclusions from the rapidly growing body of research on end-to-end autonomous driving. In this paper, we present NAVSIM, a middle ground between these evaluation paradigms, where we use large datasets in combination with a non-reactive simulator to enable large-scale real-world benchmarking. Specifically, we gather simulation-based metrics, such as progress and time to collision, by unrolling bird's eye view abstractions of the test scenes for a short simulation horizon. Our simulation is non-reactive, i.e., the evaluated policy and environment do not influence each other. As we demonstrate empirically, this decoupling allows open-loop metric computation while being better aligned with closed-loop evaluations than traditional displacement errors. NAVSIM enabled a new competition held at CVPR 2024, where 143 teams submitted 463 entries, resulting in several new insights. On a large set of challenging scenarios, we observe that simple methods with moderate compute requirements such as TransFuser can match recent large-scale end-to-end driving architectures such as UniAD. Our modular framework can potentially be extended with new datasets, data curation strategies, and metrics, and will be continually maintained to host future challenges. Our code is available at https://github.com/autonomousvision/navsim.

Subfields: Automotive Engineering, Control and Systems Engineering, Management Science and Operations Research

View Paper: https://www.semanticscholar.org/paper/a5e61c6d84fa0b558cb04f2731f80cc46d0e4016

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Automotive Engineering

In Control and Systems Engineering

In Management Science and Operations Research

Citations in the corpus

Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability (2024, positive): “As demonstrated in NAVSIM [23], using the front view alone results in only a 1 .”
End-to-End Autonomous Driving: Challenges and Frontiers (2023, methodology): “The nuPlan simulator is currently accessible for evaluating end-to-endsystemsviatheNAVSIMproject[124].”
DiffusionDrive: Truncated Diffusion Model for End-to-End Autonomous Driving (2024, positive): “We benchmark our method on the planning-oriented NAVSIM dataset [11] using non-reactive simulation and closed-loop evaluations.”