Multi-Task Multi-Sensor Fusion for 3D Object Detection

By Ming Liang, Binh Yang, Yun Chen, Rui Hu, Raquel Urtasun. Computer Vision and Pattern Recognition, 2019

687 citations recorded by the source corpus.

In this paper we propose to exploit multiple related tasks for accurate multi-sensor 3D object detection. Towards this goal we present an end-to-end learnable architecture that reasons about 2D and 3D object detection as well as ground estimation and depth completion. Our experiments show that all these tasks are complementary and help the network learn better representations by fusing information at various levels. Importantly, our approach leads the KITTI benchmark on 2D, 3D and bird's eye view object detection, while being real-time.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/8c31e0862a7f26fd1a4a52045ffcf24b8b82292e

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Disentangling Monocular 3D Object Detection (2019, methodology): “Even though there are promising methods based on multi-sensor fusion (usually exploiting LIDAR information [14, 33, 30] next to RGB images), 3D detection results produced from a single, monocular RGB input image lag considerably behind.”
SE-SSD: Self-Ensembling Single-Stage Object Detector From Point Cloud (2021, methodology): “41 65 UberATG-MMF [12] CVPR 2019 LiDAR+RGB 88.”
3D-CVF: Generating Joint Camera and LiDAR Features Using Cross-View Spatial Feature Fusion for 3D Object Detection (2020, methodology): “in through NMS post-processing with the IoU threshold 0.7. 3D RoI-based Feature Fusion: The predicted box regression values are translated to the global coordinates using the rotated 3D RoI alignment [12]. The low-level LiDAR and camera features are pooled using 3D RoI-based pooling and combined wit”
EPNet: Enhancing Point Features with Image Semantics for 3D Object Detection (2020, methodology): “52 90.99 84.82 79.62 85.14 94.65 88.61 83.71 88.99 ContFuse [18] L+I 83.68 68.78 61.67 71.38 94.07 85.35 75.88 85.10 - - - - PC-CNN [5] L+I 85.57 73.79 65.65 75.00 91.19 87.40 79.35 85.98 - - - - MMF [17] L+I 88.40 77.43 70.22 78.68 93.67 88.21 81.99 87.96 - - - - Ours L+I 89.81 79.28 74.59 81.23 94”
CLOCs: Camera-LiDAR Object Candidates Fusion for 3D Object Detection (2020, methodology): “MMF is currently one of the best public multi-modal fusion based 3D detector according to the KITTI 3D/BEV object detection benchmark.”
BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation (2022, background): “MMF [26] simultaneously works on depth completion and object detection with both camera and LiDAR inputs, but is still object-centric and not applicable to BEV map segmentation.”
Improving 3D Object Detection with Channel-wise Transformer (2021, background): “Another category of methods [3, 13, 34, 35, 28, 15, 12, 16, 17] aimed to voxelize the unstructured point clouds as a regular 2D/3D grid over which conventional CNNs can be easily applied.”
PointPainting: Sequential Fusion for 3D Object Detection (2019, methodology): “Indeed, despite recent fusion research [17, 26, 12, 32], the top methods on PointPainting is a general fusion method that be can used with any lidar detection network.”
OPV2V: An Open Benchmark Dataset and Fusion Pipeline for Perception with Vehicle-to-Vehicle Communication (2021, methodology): “With recent advancements in robotic sensing and machine learning, the reliability of perception has been signiﬁcantly improved [1], [2], [3], and 3D object detection algorithms have achieved outstanding performance either with LiDAR point clouds [4], [5], [6], [7] or multi-sensor data [8], [9].”
One Million Scenes for Autonomous Driving: ONCE Dataset (2021, background): “2) Multi-modality 100 approaches [42, 9, 21, 37, 25, 52] leverage both point clouds and images for 3D object detection.”
STD: Sparse-to-Dense 3D Object Detector for Point Cloud (2019, methodology): “Compared to UberATG-MMF [19], which is the best multi-sensor detector, STD outperforms it by 0.88% on the moderate level on 3D detection of Cars.”
PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection (2019, methodology): “The pioneer work MV3D [1] projects the point clouds to 2D bird view grids and places lots of predefined 3D anchors for generating 3D bounding boxes, and the following works [12, 18, 17] develop better strategies for multi-sensor fusion while [39, 38, 13] propose more efficient frameworks with bird v”
CenterFusion: Center-based Radar and Camera Fusion for 3D Object Detection (2020, background): “Sensor fusion is one of these challenges, which has motivated many studies in 2D and 3D object detection [4, 10, 14, 19], semantic segmentation [33, 16] and object tracking [1, 7] in recent years.”
One Thousand and One Hours: Self-driving Motion Prediction Dataset (2020, background): “In the ﬁeld of self-driving vehicles (SDV), several datasets, such as [9, 6, 13] enabled great progress in the development of perception systems [14, 16, 24, 20].”
PointAugmenting: Cross-Modal Augmentation for 3D Object Detection (2021, background): “Researchers [11, 10, 29] have attempted to transform the front-view camera features into BEV maps.”
Vision-based robotic grasping from object localization, object pose estimation to grasp estimation for parallel grippers: a review (2020, methodology): “, 2018; Wang and Jia, 2019], global regressionbased methods [Song and Xiao, 2016; Chen et al., 2017; Liang et al., 2019b] and local regression-based methods.”
TransFuser: Imitation With Transformer-Based Sensor Fusion for Autonomous Driving (2022, methodology): “They operate primarily based on geometric feature projections between the image space and different LiDAR projection spaces, e.g. Bird’s Eye View (BEV) [14], [15], [16], [17], [18], [19], [20], [21], [22] and Range View (RV) [14], [17], [23], [30], [34], [35].”