LiDARsim: Realistic LiDAR Simulation by Leveraging the Real World

By Sivabalan Manivasagam, Shenlong Wang, K. Wong, Wenyuan Zeng, Mikita Sazanovich, Shuhan Tan, Binh Yang, Wei-Chiu Ma, Raquel Urtasun. Computer Vision and Pattern Recognition, 2020

300 citations recorded by the source corpus.

We tackle the problem of producing realistic simulations of LiDAR point clouds, the sensor of preference for most self-driving vehicles. We argue that, by leveraging real data, we can simulate the complex world more realistically compared to employing virtual worlds built from CAD/procedural models. Towards this goal, we first build a large catalog of 3D static maps and 3D dynamic objects by driving around several cities with our self-driving fleet. We can then generate scenarios by selecting a scene from our catalog and "virtually" placing the self-driving vehicle (SDV) and a set of dynamic objects from the catalog in plausible locations in the scene. To produce realistic simulations, we develop a novel simulator that captures both the power of physics-based and learning-based simulation. We first utilize raycasting over the 3D scene and then use a deep neural network to produce deviations from the physics-based simulation, producing realistic LiDAR point clouds. We showcase LiDARsim's usefulness for perception algorithms-testing on long-tail events and end-to-end closed-loop evaluation on safety-critical scenarios.

Subfields: Artificial Intelligence, Automotive Engineering, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/e7399a05c1ae2f70558fcf1a3dcfc7a06a7ae6c9

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Automotive Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

V2X-Sim: Multi-Agent Collaborative Perception Dataset and Benchmark for Autonomous Driving (2022, background): “The most relevant work is V2V-Sim [12], which is based on a high-quality LiDAR simulator [26].”
3D Object Detection for Autonomous Driving: A Comprehensive Survey (2022, background): “Some approaches (Manivasagam et al., 2020) leverage real-world instances to reconstruct 3D scenes.”
Evaluating Real-World Robot Manipulation Policies in Simulation (2024, unknown): “In this section, we outline our problem definition and discuss metrics for measuring the quality of simulated evaluation pipelines.”
V2V4Real: A Real-World Large-Scale Dataset for Vehicle-to-Vehicle Cooperative Perception (2023, background): “However, it is well known that there exists a clear domain gap between synthetic data and real-world data, as the traffic behavior and sensor rendering in simulators are often not realistic enough [27, 37].”
Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecasting (2023, background): “In turn, these improvements have made isolation of object-specific point clouds possible, which has proven invaluable for offboard detection and tracking [37], and for simulation [8], which previously required human-annotated 3D bounding boxes [30].”
V2VNet: Vehicle-to-Vehicle Communication for Joint Perception and Prediction (2020, methodology): “d contribution is a new dataset, dubbed V2V-Sim (see Fig. 1, right) that mimics the setting where there are multiple SDVs driving in the area. Towards this goal, we use a high-delity LiDAR simulator [33], which uses a large catalog of static 3D scenes and dynamic objects built from real-world data, ”
MP3: A Unified Model to Map, Perceive, Predict and Plan (2021, background): “Closed-loop Simulation Results: Our simulated environment leverages a state-of-the-art LiDAR simulator [32] to recreate a virtual world from previously collected real static environments and a large-scale bank of diverse actors.”
TrafficSim: Learning to Simulate Realistic Multi-Agent Behaviors (2021, methodology): “Real world data is leveraged by [32] for realistic sensor simulation.”
UniSim: A Neural Closed-Loop Sensor Simulator (2023, methodology): “LiDAR Simulation: We also evaluate the fidelity of our LiDAR simulation and compare with SoTA approach LiDARsim [46].”
OPV2V: An Open Benchmark Dataset and Fusion Pipeline for Perception with Vehicle-to-Vehicle Communication (2021, methodology): “[15] utilizes a high-fidelity LiDAR simulator [23] to generate a large-scale V2V dataset.”
Recent Advancements in End-to-End Autonomous Driving Using Deep Learning: A Survey (2023, unknown): “The details are provided in Table VI.”
A Review and Comparative Study on Probabilistic Object Detection in Autonomous Driving (2020, background): “In this case, building autonomous vehicle simulators with high ﬁdelity could help (e.g. [122], [123]).”