End-to-End Multi-View Fusion for 3D Object Detection in LiDAR Point Clouds

By Yin Zhou, Pei Sun, Yu Zhang, Dragomir Anguelov, Jiyang Gao, Tom Y. Ouyang, James Guo, Jiquan Ngiam, Vijay Vasudevan. Conference on Robot Learning, 2019

401 citations recorded by the source corpus.

Recent work on 3D object detection advocates point cloud voxelization in birds-eye view, where objects preserve their physical dimensions and are naturally separable. When represented in this view, however, point clouds are sparse and have highly variable point density, which may cause detectors difficulties in detecting distant or small objects (pedestrians, traffic signs, etc.). On the other hand, perspective view provides dense observations, which could allow more favorable feature encoding for such cases. In this paper, we aim to synergize the birds-eye view and the perspective view and propose a novel end-to-end multi-view fusion (MVF) algorithm, which can effectively learn to utilize the complementary information from both. Specifically, we introduce dynamic voxelization, which has four merits compared to existing voxelization methods, i) removing the need of pre-allocating a tensor with fixed size; ii) overcoming the information loss due to stochastic point/voxel dropout; iii) yielding deterministic voxel embeddings and more stable detection outcomes; iv) establishing the bi-directional relationship between points and voxels, which potentially lays a natural foundation for cross-view feature fusion. By employing dynamic voxelization, the proposed feature fusion architecture enables each point to learn to fuse context information from different views. MVF operates on points and can be naturally extended to other approaches using LiDAR point clouds. We evaluate our MVF model extensively on the newly released Waymo Open Dataset and on the KITTI dataset and demonstrate that it significantly improves detection accuracy over the comparable single-view PointPillars baseline.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/b63b861cb907331dfd5d7dbfbb0a1563f25f2c5a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

DeepFusion: Lidar-Camera Deep Fusion for Multi-Modal 3D Object Detection (2022, methodology): “Different from standard implementation of attention module, our implementation is combined with dynamic voxelization [45].”
StarNet: Targeted Computation for Object Detection in Point Clouds (2019, methodology): “…Multi View Fusion [52] 65.3 62.9 StarNet 66.8 53.7 Table 3: Waymo Open Dataset Validation set results of LEVEL 1 difﬁculty for comparison with Multi-View Fusion [52] and a reimplemented PointPillars [22]. of 1 for both vehicles and pedestrian models, as it exhibits substantially higher performance.”
Not All Points Are Equal: Learning Highly Efficient Point-based Detectors for 3D LiDAR Point Clouds (2022, neutral): “Due to the unstructured and orderless nature of 3D point clouds, early works usually first convert the raw point clouds into intermediate regular representation, including projecting the 3D point clouds into 2D images from birdseye-view or frontal view [1, 17, 18, 41, 42, 50, 60], or transformed int”
Multi-Modal Fusion Transformer for End-to-End Autonomous Driving (2021, background): “object detection [22, 12, 66, 7, 44, 31, 34, 61, 33, 37] and motion forecasting [36, 5, 35, 63, 6, 19, 38].”
PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection (2019, methodology): “Table 5 shows that our method outperforms previous state-of-theart [44] significantly with a 7.”
MP3: A Unified Model to Map, Perceive, Predict and Plan (2021, background): “Perception and Prediction: Most previous works perform object detection [15, 25, 34, 52, 55] and actor-based prediction to reason about the current and future state of a driving scene.”
Recent Advancements in End-to-End Autonomous Driving Using Deep Learning: A Survey (2023, background): “Zhou et al. [57] perform information fusion at the mid-level by leveraging the complementary information provided by both the bird’s-eye view (BEV) and perspective views of the LiDAR point cloud.”
Scalability in Perception for Autonomous Driving: Waymo Open Dataset (2019, methodology): “We also use dynamic voxelization [24] where every location that has points is voxelized instead of having a fixed number of pillars.”
HDMapNet: An Online HD Map Construction and Evaluation Framework (2021, methodology): “For point clouds, we use a variant of PointPillars [39] with dynamic voxelization [31].”
Center-based 3D Object Detection and Tracking (2020, background): “MVF [65] and Pillar-od [52] combine multiple view features to learn a more effective pillar representation.”
Towards Robust LiDAR-based Perception in Autonomous Driving: General Black-box Adversarial Sensor Attack and Countermeasures (2020, methodology): “The reproduced results also align well with the evaluations in [24, 72].”
Improving 3D Object Detection with Channel-wise Transformer (2021, methodology): “Waymo dataset consists of 798 training sequences with around 158,361 LiDAR samples, and 202 validation sequences with 40,077 LiDAR samples.”
VectorMapNet: End-to-end Vectorized HD Map Learning (2022, methodology): “For LiDAR data P , we use a variant of PointPillars (Lang et al., 2019) with dynamic voxelization (Zhou et al., 2020), which divides the 3D space into multiple pillars and uses pillar-wise point clouds to learn pillar-wise feature maps.”
BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation (2022, methodology): “Researchers have designed single-stage 3D object detectors [72, 21, 65, 73, 66, 71] that extract flattened point cloud features using PointNets [41] or SparseConvNet [17] and perform detection in the BEV space.”
RPVNet: A Deep and Efficient Range-Point-Voxel Fusion Network for LiDAR Point Cloud Segmentation (2021, methodology): “Compared to other previous multi-view fusion methods [6, 42, 33, 20, 39, 14], which either fuse in the front or the end of network, our method conducts aforementioned fusion multi-times in the network, which allows different views to enhance each other in a deeper and more flexible way.”