MVX-Net: Multimodal VoxelNet for 3D Object Detection

By Vishwanath A. Sindagi, Yin Zhou, Oncel Tuzel. IEEE International Conference on Robotics and Automation, 2019

546 citations recorded by the source corpus.

Many recent works on 3D object detection have focused on designing neural network architectures that can consume point cloud data. While these approaches demonstrate encouraging performance, they are typically based on a single modality and are unable to leverage information from other modalities, such as a camera. Although a few approaches fuse data from different modalities, these methods either use a complicated pipeline to process the modalities sequentially, or perform late-fusion and are unable to learn interaction between different modalities at early stages. In this work, we present PointFusion and VoxelFusion: two simple yet effective early-fusion approaches to combine the RGB and point cloud modalities, by leveraging the recently introduced VoxelNet architecture. Evaluation on the KITTI dataset demonstrates significant improvements in performance over approaches which only use point cloud data. Furthermore, the proposed method provides results competitive with the state-of-the-art multimodal algorithms, achieving top-2 ranking in five of the six birds eye view and 3D detection categories on the KITTI benchmark, by using a simple single stage network.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Geology

View Paper: https://www.semanticscholar.org/paper/dafd2826ec24d0dbdf0f4df7b6f20e7e2448f802

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In Geology

Citations in the corpus

DAIR-V2X: A Large-Scale Dataset for Vehicle-Infrastructure Cooperative 3D Object Detection (2022, methodology): “…a) In early fusion, raw data from different sensors are directly transferred and fused [9]. b) In intermediate fusion, intermediate representations like features extracted from the models are fused [4,21]. c) In late fusion, the prediction outputs like 3D information of the objects are fused [11].”
Deep Multi-Modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges (2019, background): “In this regard, some papers including [98], [102]–[105], [107], [113], [115] combine RGB camera images and LiDAR point clouds for 3D object detection.”
BEVFusion: A Simple and Robust LiDAR-Camera Fusion Framework (2022, methodology): “Specifically, the majority of works can be summarized as follow: i) given one or a few points of the LiDAR point cloud, LiDAR to world transformation matrix and the essential matrix (camera to world); ii) people transform the LiDAR points [41, 44, 45, 44, 16, 57] or proposals Corresponding Author.”
Sensor and Sensor Fusion Technology in Autonomous Vehicles: A Review (2021, methodology): “Reference [191] leverages their previously proposed VoxelNet framework in [192] and presented two feature-level fusion approaches called PointFusion and VoxelFusion to combine the RGB and point cloud data for 3D object detection.”
One Million Scenes for Autonomous Driving: ONCE Dataset (2021, background): “2) Multi-modality approaches [43, 9, 22, 38, 26, 53] leverage both point clouds and images for 3D object detection.”
TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers (2022, background): “Since PointPainting [46] was proposed, the point-level fusion methods [10,40,47] have shown great advantages and promising results.”
Seeing Through Fog Without Seeing Fog: Deep Multimodal Sensor Fusion in Unseen Adverse Weather (2019, background): “As large automotive datasets [66, 19, 59] cover limited sensory inputs, existing fusion methods have been proposed mostly for lidar-camera setups [65, 56, 11, 36, 43].”
Deep Learning for Image and Point Cloud Fusion in Autonomous Driving: A Review (2020, unknown): “MVX-Net [71] presented by Sindagi et al.”
End-to-End Multi-View Fusion for 3D Object Detection in LiDAR Point Clouds (2019, background): “Several approaches have looked at combining BEV laser data with perspective RGB images, either at the ROI pooling stage (MV3D [5], AVOD [6]) or at a per-point level (MVX-Net [7]).”
PointAugmenting: Cross-Modal Augmentation for 3D Object Detection (2021, methodology): “Our method differs from the prior 3D detector MVX-Net [16], which utilizes the high-level semantics on the Conv5 layer of VGG-16.”