PointAugmenting: Cross-Modal Augmentation for 3D Object Detection

By Chunwei Wang, Chao Ma, Ming Zhu, Xiaokang Yang. Computer Vision and Pattern Recognition, 2021

485 citations recorded by the source corpus.

Camera and LiDAR are two complementary sensors for 3D object detection in the autonomous driving context. Camera provides rich texture and color cues while LiDAR specializes in relative distance sensing. The challenge of 3D object detection lies in effectively fusing 2D camera images with 3D LiDAR points. In this paper, we present a novel cross-modal 3D object detection algorithm, named PointAugmenting. On one hand, PointAugmenting decorates point clouds with corresponding point-wise CNN features extracted by pretrained 2D detection models, and then performs 3D object detection over the decorated point clouds. In comparison with highly abstract semantic segmentation scores to decorate point clouds, CNN features from detection networks adapt to object appearance variations, achieving significant improvement. On the other hand, PointAugmenting benefits from a novel cross-modal data augmentation algorithm, which consistently pastes virtual objects into images and point clouds during network training. Extensive experiments on the large-scale nuScenes and Waymo datasets demonstrate the effectiveness and efficiency of our PointAugmenting. Notably, PointAugmenting outperforms the LiDAR-only baseline detector by +6.5% mAP and achieves the new state-of-the-art results on the nuScenes leaderboard to date.

Subfields: Aerospace Engineering, Automotive Engineering, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/747a2bbe3765f488eedac33a2ac7a0069020da57

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

1. Or Litany
2. Jiaya Jia
3. Song Han
4. Jian‐Yuan Sun
5. Quoc V. Le
6. Zhijian Liu
7. X. Zhang
8. Xinge Zhu

In Automotive Engineering

1. Alexander Amini
2. Daniela Rus
3. Alan L. Yuille
4. Jiaya Jia
5. Jiquan Ngiam
6. Yilun Chen
7. Quoc V. Le

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Alan L. Yuille
3. Dacheng Tao
4. Xiaogang Wang
5. Ming-Hsuan Yang
6. Trevor Darrell
7. Quoc V. Le
8. Yi Yang

Citations in the corpus

• DeepFusion: Lidar-Camera Deep Fusion for Multi-Modal 3D Object Detection (2022, methodology): “Specifically, the methods we consider are (1) InputFusion, that fuses the camera features with the lidar points at the input stage [34, 36], (2) LateFusion, where the lidar points and camera features go through voxelizer separately, followed by a concatenation [36], and (3) our proposed DeepFusion.”
• BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation (2022, methodology): “They augment the LiDAR point cloud with semantic labels [54], CNN features [55, 23] or virtual points from 2D images [68], and then apply”
• BEVFusion: A Simple and Robust LiDAR-Camera Fusion Framework (2022, methodology): “Previous fusion methods can be broadly categorized into (a) point-level fusion mechanism [43, 46, 47, 46, 17, 60] that project image features onto raw point clouds, and (b) feature-level fusion mechanism [4, 61, 1, 21] that projects LiDAR feature or proposals on each view image separately to…”
• Focal Sparse Convolutional Networks for 3D Object Detection (2022, methodology): “For these invertible transformations, we reverse the coordinates of sparse features with the recorded transformation parameters [44,57].”
• TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers (2022, methodology): “Since Point-Painting [47] was proposed, the point-level fusion meth-ods [10,40,48] have shown great advantages and promising results.”
• Language-Grounded Indoor 3D Semantic Segmentation in the Wild (2022, methodology): “Various methods have been introduced towards data-sampling-based re-balancing, for instance random oversampling of underrepresented classes [3,54,50], sampling novel poses of known categories [32], undersampling overrepresented classes [35], frequency-based sampling [25], as well as feature-based…”