Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D

By Jonah Philion, Sanja Fidler. European Conference on Computer Vision, 2020

1,686 citations recorded by the source corpus.

The goal of perception for autonomous vehicles is to extract semantic representations from multiple sensors and fuse these representations into a single "bird's-eye-view" coordinate frame for consumption by motion planning. We propose a new end-to-end architecture that directly extracts a bird's-eye-view representation of a scene given image data from an arbitrary number of cameras. The core idea behind our approach is to "lift" each image individually into a frustum of features for each camera, then "splat" all frustums into a rasterized bird's-eye-view grid. By training on the entire camera rig, we provide evidence that our model is able to learn not only how to represent images but how to fuse predictions from all cameras into a single cohesive representation of the scene while being robust to calibration error. On standard bird's-eye-view tasks such as object segmentation and map segmentation, our model outperforms all baselines and prior work. In pursuit of the goal of learning dense representations for motion planning, we show that the representations inferred by our model enable interpretable end-to-end motion planning by "shooting" template trajectories into a bird's-eye-view cost map output by our network. We benchmark our approach against models that use oracle depth from lidar. Project page with code: this https URL .

View Paper: https://www.semanticscholar.org/paper/4def603277e077d11eb6ce030df931330d758207

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

VADv2: End-to-End Vectorized Autonomous Driving via Probabilistic Planning (2024, background): “LSS [38] is a pioneering work that achieves the perspective view to BEV transformation by explicitly predicting depth for image pixels.”
TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers (2022, methodology): “We believe that our method could benefit from more research progress [26, 32, 33] in this direction.”
MP3: A Unified Model to Map, Perceive, Predict and Plan (2021, methodology): “This is in contrast to the simplistic approach of, e.g., [37] where a ﬁxed set of template trajectories is used, ignoring the initial state of SDV.”
Cross-view Transformers for real-time Map-view Semantic Segmentation (2022, methodology): “Using the pose of the ego-vehicle, we generate the ground-truth labels y, a binary vehicle occupancy mask rendered at a resolution of (200, 200) by orthographically projecting 3D box annotations onto the ground plane, following standard practice [14, 29].”
BEVDet4D: Exploit Temporal Cues in Multi-camera 3D Object Detection (2022, methodology): “A more precise method is to adjust the coordinates of the point cloud generated by the lifting operation in the view trans-former [30].”
NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenario (2023, methodology): “Then, in order to make the feature spatial-aware, we estimate the depth of the 2D points in the images and lift them to 3D virtual points with a view transformer inspired by LSS [34].”
Unifying Voxel-based Representation with Transformer for 3D Object Detection (2022, methodology): “Motivated by [39, 5], we first generate the depth distribution DI ∈ RD×H×W of each image with a single convolution as DI(u, v) = Softmax(Conv(FI)(u, v)).”
Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Driving (2023, unknown): “More recently, vision-based 3D object detection has gained more attention due to its low cost and rich semantic content [41, 46, 48, 22, 26, 33, 28, 13, 27, 15, 21, 31, 25].”
ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learning (2022, methodology): “However, the evolution process of the past is not well captured and exploited [25, 39, 47].”
MapTR: Structured Modeling and Learning for Online Vectorized HD Map Construction (2022, methodology): “And for fair comparison with IPM and LSS, GKT and Deformable Attention both adopt one-layer conﬁguration.”
PETR: Position Embedding Transformation for Multi-View 3D Object Detection (2022, methodology): “Similar to CaDDN [38], BEVDet [18] employs the Lift-Splat-Shoot [37] to transform 2D multi-view features into BEV representation.”
Categorical Depth Distribution Network for Monocular 3D Object Detection (2021, methodology): “Depth information in image data can also be learned implicitly, by directly transforming features from images to 3D space and finally to bird’s-eye-view (BEV) grids [48, 44].”
VAD: Vectorized Scene Representation for Efficient Autonomous Driving (2023, methodology): “Previous works [19, 21] use rasterized semantic maps to guide the planning, which misses critical instance-level structure information of the map.”
BEVDepth: Acquisition of Reliable Depth for Multi-view 3D Object Detection (2022, methodology): “Benefiting from the decoupled nature of LSS [3], the camera-aware depth prediction module is isolated from the detection head and thus the regression target in this case does not need to be changed, resulting in greater extensibility.”
BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation (2022, methodology): “For each frame, we only perform the evaluation in the [-50m, 50m]×[-50m, 50m] region around the ego car following [34, 64, 58, 22].”
HDMapNet: An Online HD Map Construction and Evaluation Framework (2021, methodology): “Lift-Splat-Shoot [29] estimates a distribution over depth in the perspective view images.”
PETRv2: A Unified Framework for 3D Perception from Multi-Camera Images (2022, methodology): “Backbone Drive Lane Vehicle Lift-Splat [33] Res-101 0.729 0.200 0.321 FIERY [10] Res-101 - - 0.382 M 2 BEV [40] X-101 0.759 0.380 - BEVFormer [20] Res-101 0.801 0.257 0.448 PETRv2 Res-101 0.833 0.448 0.434 PETRv2 ∗ V2-99 0.856 0.490 0.463 Table 4.”
Tri-Perspective View for Vision-Based 3D Semantic Occupancy Prediction (2023, methodology): “Recent studies in BEV-based perception focus on how to effectively transform features from the image space to the BEV space [20, 26, 27, 35, 36, 48].”