PointPainting: Sequential Fusion for 3D Object Detection

By Sourabh Vora, Alex H. Lang, Bassam Helou, Oscar Beijbom. Computer Vision and Pattern Recognition, 2019

1,098 citations recorded by the source corpus.

Camera and lidar are important sensor modalities for robotics in general and self-driving cars in particular. The sensors provide complementary information offering an opportunity for tight sensor-fusion. Surprisingly, lidar-only methods outperform fusion methods on the main benchmark datasets, suggesting a gap in the literature. In this work, we propose PointPainting: a sequential fusion method to fill this gap. PointPainting works by projecting lidar points into the output of an image-only semantic segmentation network and appending the class scores to each point. The appended (painted) point cloud can then be fed to any lidar-only method. Experiments show large improvements on three different state-of-the art methods, Point-RCNN, VoxelNet and PointPillars on the KITTI and nuScenes datasets. The painted version of PointRCNN represents a new state of the art on the KITTI leaderboard for the bird's-eye view detection task. In ablation, we study how the effects of Painting depends on the quality and format of the semantic segmentation output, and demonstrate how latency can be minimized through pipelining.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Instrumentation

View Paper: https://www.semanticscholar.org/paper/8648f59b62dda3900d597e944abe8af51a18f665

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In Instrumentation

Citations in the corpus

One Million Scenes for Autonomous Driving: ONCE Dataset (2021, methodology): “PointPainting [42] appends the segmentation scores to the 233 input point clouds of CenterPoints [51], but the performance drops from 61.”
Focal Sparse Convolutional Networks for 3D Object Detection (2022, methodology): “Second, we only strengthen the predicted important features, instead of the uniform decoration [43, 44] for all LIDAR features.”
An End-to-End Transformer Model for 3D Object Detection (2021, methodology): “We build upon the Transformer [68] which is applicable for sets but not tailored for 3D. 3D Object Detection is a well studied research area where methods predict three dimensional bounding boxes from 3D input data [23, 41, 43, 52, 54, 55, 70, 72, 93].”
TransFuser: Imitation With Transformer-Based Sensor Fusion for Autonomous Driving (2022, neutral): “Most sensor fusion works consider perception tasks, e.g. object detection [14], [15], [16], [18], [19], [20], [21], [22], [23], [47], [48], [49], [50], [51], [52], [53], [54], [55], [56], [57], [58], [59], [60] and motion forecasting [24], [25], [26], [27], [28], [29], [30], [49], [61], [62].”
PandaSet: Advanced Sensor Suite Dataset for Autonomous Driving (2021, methodology): “To establish the baseline for LiDAR-camera fusion 3D object detection, we re-implement PointPainting [21], an effective sequential architecture to fuse point clouds with semantic information from images.”
BEVFusion: A Simple and Robust LiDAR-Camera Fusion Framework (2022, methodology): “Previous fusion methods can be broadly categorized into (a) point-level fusion mechanism [43, 46, 47, 46, 17, 60] that project image features onto raw point clouds, and (b) feature-level fusion mechanism [4, 61, 1, 21] that projects LiDAR feature or proposals on each view image separately to…”
BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation (2022, methodology): “They augment the LiDAR point cloud with semantic labels [54], CNN features [55, 23] or virtual points from 2D images [68], and then apply”
V2V4Real: A Real-World Large-Scale Dataset for Vehicle-to-Vehicle Cooperative Perception (2023, methodology): “Some methods [29, 39] use a two-step framework, e.”
Deep Learning for Image and Point Cloud Fusion in Autonomous Driving: A Review (2020, background): “To be more specific, PointPainting [66] follows the idea of projecting points to 2D semantic maps in [61].”
3D Object Detection with Pointformer (2020, methodology): “Detectors are designed either with point clouds as the only input [42, 41, 15, 39, 24, 25, 19, 26, 35, 40] or fusing multiple sensor modalities such as LiDAR and camera [20, 17, 31].”
DeepFusion: Lidar-Camera Deep Fusion for Multi-Modal 3D Object Detection (2022, methodology): “Specifically, the methods we consider are (1) InputFusion, that fuses the camera features with the lidar points at the input stage [34, 36], (2) LateFusion, where the lidar points and camera features go through voxelizer separately, followed by a concatenation [36], and (3) our proposed DeepFusion.”
PointAugmenting: Cross-Modal Augmentation for 3D Object Detection (2021, methodology): “Seg Score: decorating LiDAR points with segmentation scores as suggested by PointPainting [19].”
Cross-view Transformers for real-time Map-view Semantic Segmentation (2022, methodology): “Psuedo-lidar based approaches [24, 44, 45, 46] project to 3D points using a depth estimate and leverage 3D point based architectures (e.g. [18, 30, 43]) with 2D labels.”
FUTR3D: A Unified Sensor Fusion Framework for 3D Detection (2022, methodology): “For example, FUTR3D outperforms PointPainting [28] without bells and whistles even though PointPainting is designed to work on high-resolution LiDARs and images.”
Multimodal Virtual Point 3D Detection (2021, methodology): “The baselines include both the Lidar-only two-stage CenterPoint [66] model and the state-of-the-art multi-modal fusion method PointPainting [52].”
3D Object Detection for Autonomous Driving: A Survey (2021, background): “(3) multimodal fusion based [12, 30, 31, 32, 33, 34], which currently lags behind its point cloud based counterparts but importantly provides a redundancy to fallback onto in case of a malfunction or outage.”
DAIR-V2X: A Large-Scale Dataset for Vehicle-Infrastructure Cooperative 3D Object Detection (2022, background): “The success of deep neural networks brings the promise of solving AD’s core requirement to perceive the surrounding environment from point cloud [15,20,28], images [7, 18] or multi-modality data [21, 24].”