PointFusion: Deep Sensor Fusion for 3D Bounding Box Estimation

By Danfei Xu, Dragomir Anguelov, Ashesh Jain. 2017

739 citations recorded by the source corpus.

We present PointFusion, a generic 3D object detection method that leverages both image and 3D point cloud information. Unlike existing methods that either use multistage pipelines or hold sensor and dataset-specific assumptions, PointFusion is conceptually simple and application-agnostic. The image data and the raw point cloud data are independently processed by a CNN and a PointNet architecture, respectively. The resulting outputs are then combined by a novel fusion network, which predicts multiple 3D box hypotheses and their confidences, using the input 3D points as spatial anchors. We evaluate PointFusion on two distinctive datasets: the KITTI dataset that features driving scenes captured with a lidar-camera setup, and the SUN-RGBD dataset that captures indoor environments with RGB-D cameras. Our model is the first one that is able to perform better or on-par with the state-of-the-art on these diverse datasets without any dataset-specific model tuning.

Subfields: Aerospace Engineering, Environmental Engineering, Geology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Environmental Engineering

In Geology

Citations in the corpus

nuScenes: A Multimodal Dataset for Autonomous Driving (2019, background): “And while there are several works that have proposed fusion methods based on cameras and lidar [48, 14, 64, 52, 81, 75, 29], PointPillars [51] showed a lidar-only method that performed on par with existing fusion based methods.”
Deep Learning for Image and Point Cloud Fusion in Autonomous Driving: A Review (2020, background): “2) Multi-level Fusion: Another possible direction of improvement is to combine result level fusion with feature level fusion, where one such work is PointFusion [62].”
Learning Object Bounding Boxes for 3D Instance Segmentation on Point Clouds (2019, methodology): “Detection performance is further improved by fusing RGB images in [3; 54; 36; 52].”
TransFuser: Imitation With Transformer-Based Sensor Fusion for Autonomous Driving (2022, neutral): “Most sensor fusion works consider perception tasks, e.g. object detection [14], [15], [16], [18], [19], [20], [21], [22], [23], [47], [48], [49], [50], [51], [52], [53], [54], [55], [56], [57], [58], [59], [60] and motion forecasting [24], [25], [26], [27], [28], [29], [30], [49], [61], [62].”
PointRCNN: 3D Object Proposal Generation and Detection From Point Cloud (2018, methodology): “[25, 39] utilized mature 2D detectors to generate 2D proposals from images and reduced the size of 3D points in each cropped image regions.”
Generating 3D Adversarial Point Clouds (2018, methodology): “We choose PointNet because the model and its variants have been widely and successfully adopted in many applications such as 3D object detection for autonomous driving [18, 34, 30], semantic segmentation for indoor scene understanding [12, 19], and AI-assisted shape design [24].”
Frustum ConvNet: Sliding Frustums to Aggregate Local Point-Wise Features for Amodal (2019, neutral): “5 6.4 15.4 53.5 50.3 78.9 42.1 COG [29] 58.26 63.67 31.80 62.17 45.19 15.47 27.36 51.02 51.29 70.07 47.63 2Ddriven3D [30] 43.45 64.48 31.40 48.27 27.93 25.92 41.92 50.39 37.02 80.40 45.12 PointFusion [31] 37.26 68.57 37.69 55.09 17.16 23.95 32.33 53.83 31.03 83.80 45.38 Ren et al. [32] 76.2 73.2 32.”
End-to-End Multi-View Fusion for 3D Object Detection in LiDAR Point Clouds (2019, methodology): “A separate line of work, such as Frustum PointNet [24] and PointFusion [25], ﬁrst generates 2D object proposals from the RGB image using a standard image detector and extrudes each 2D detection box to a 3D frustum, which is then processed by a PointNet-like network[22, 26] to predict the correspondi”
6-DOF GraspNet: Variational Grasp Generation for Object Manipulation (2019, background): “The success of PointNet lead to the introduction of different variations of network architectures [33, 30] that represent 3D data, showing significant improvement on 3D object pose estimation, semantic segmentation, and part segmentation [30, 24, 22, 34].”
Deep Learning for LiDAR Point Clouds in Autonomous Driving: A Review (2020, methodology): “The second is that the existing data fusion scheme using DL knowledge is processed in a separate line, which is not an end-to-end scheme [41], [119], [134].”
EC-Net: an Edge-aware Point set Consolidation Network (2018, background): “The network was trained with an edge-aware joint loss function that has four terms; see the yellow boxes on the right. for high-level analysis problems with point clouds [31,32,33,34,35,36,37,38,39,40].”
CenterFusion: Center-based Radar and Camera Fusion for 3D Object Detection (2020, methodology): “PointFusion [28] processes the image and LiDAR data using a CNN and a PointNet model respectively, and then generate 3D object proposals using the extracted features.”
LaserNet: An Efficient Probabilistic 3D Object Detector for Autonomous Driving (2019, methodology): “In addition to using LiDAR data, several previous works use images [5, 15, 22, 17, 26, 21, 9] or high definition maps [27] to improve detections.”
DenseFusion: 6D Object Pose Estimation by Iterative Dense Fusion (2019, methodology): “6D object pose estimation is the crux to many important real-world applications, such as robotic grasping and manipulation [7, 35, 44], autonomous navigation [6, 11, 42], and augmented reality [19, 20].”
Seeing Through Fog Without Seeing Fog: Deep Multimodal Sensor Fusion in Unseen Adverse Weather (2019, methodology): “Multisensor Fusion Multisensor feeds in autonomous vehicles are typically fused to exploit varying cues in the measurements [44], simplify path-planning [15], to allow for redundancy in the presence of distortions [48], or solve for joint vision tasks, such as 3D object detection [65].”