3D-CVF: Generating Joint Camera and LiDAR Features Using Cross-View Spatial Feature Fusion for 3D Object Detection

By Jin Hyeok Yoo, Yecheol Kim, Ji Song Kim, Jun Won Choi. European Conference on Computer Vision, 2020

511 citations recorded by the source corpus.

In this paper, we propose a new deep architecture for fusing camera and LiDAR sensors for 3D object detection. Because the camera and LiDAR sensor signals have different characteristics and distributions, fusing these two modalities is expected to improve both the accuracy and robustness of 3D object detection. One of the challenges presented by the fusion of cameras and LiDAR is that the spatial feature maps obtained from each modality are represented by significantly different views in the camera and world coordinates; hence, it is not an easy task to combine two heterogeneous feature maps without loss of information. To address this problem, we propose a method called 3D-CVF that combines the camera and LiDAR features using the cross-view spatial feature fusion strategy. First, the method employs auto-calibrated projection, to transform the 2D camera features to a smooth spatial feature map with the highest correspondence to the LiDAR features in the bird's eye view (BEV) domain. Then, a gated feature fusion network is applied to use the spatial attention maps to mix the camera and LiDAR features appropriately according to the region. Next, camera-LiDAR feature fusion is also achieved in the subsequent proposal refinement stage. The camera feature is used from the 2D camera-view domain via 3D RoI grid pooling and fused with the BEV feature for proposal refinement. Our evaluations, conducted on the KITTI and nuScenes 3D object detection datasets demonstrate that the camera-LiDAR fusion offers significant performance gain over single modality and that the proposed 3D-CVF achieves state-of-the-art performance in the KITTI benchmark.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/2d4741e76aa6d5b3b6586b1fc69d47f9cfd8c3ab

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

One Million Scenes for Autonomous Driving: ONCE Dataset (2021, background): “2) Multi-modality 100 approaches [42, 9, 21, 37, 25, 52] leverage both point clouds and images for 3D object detection.”
Sensor and Sensor Fusion Technology in Autonomous Vehicles: A Review (2021, methodology): “Reference [178] proposed a two-stage 3D obstacle detection architecture, named 3D-cross view fusion (3D-CVF).”
BEVFusion: A Simple and Robust LiDAR-Camera Fusion Framework (2022, background): “…points and then concatenates them back as additional point features [17, 43, 47, 60]; (b) feature-level fusion where one ﬁrstly projects the LiDAR points into a feature space [61] or generates proposals [1], queries the associated camera features then concatenates back to the feature space [24, 6].”
SE-SSD: Self-Ensembling Single-Stage Object Detector From Point Cloud (2021, methodology): “3D-CVF [33] obtains semantics from multi-view images and fuses them with point features in both stages, whereas CLOCs PVCas [19] fuses semantic features from images and points to refine the predicted confidence.”
PV-RCNN++: Point-Voxel Feature Set Abstraction With Local Vector Representation for 3D Object Detection (2021, background): “99 - - - - 3D-CVF at SPA [57] ECCV 2020 RGB + LiDAR 89.”
3D Object Detection for Autonomous Driving: A Comprehensive Survey (2022, background): “Themulti-modal fusion can also be conducted only at the output featuremaps of backbone networks, with fusion modules and operators including gated attention (Yoo et al., 2020), unified object queries (Chen et al.”
Unifying Voxel-based Representation with Transformer for 3D Object Detection (2022, background): “And instance-level fusion [33, 34, 35] is usually applied at the later stage to combine object-level features.”
TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers (2022, background): “Similarly, [16, 17, 50, 59] first project a point cloud onto the bird’s eye view (BEV) plane and then fuse the image features with the BEV pixels.”
Multimodal Virtual Point 3D Detection (2021, unknown): “Multi-modal 3D Object Detection fuses information of Lidar and color cameras [20, 28, 28, 29, 35, 37, 53, 67, 71, 72].”
PointAugmenting: Cross-Modal Augmentation for 3D Object Detection (2021, methodology): “Researchers [11, 10, 29] have attempted to transform the front-view camera features into BEV maps.”
Focal Sparse Convolutional Networks for 3D Object Detection (2022, methodology): “Multi-modal fusion methods [19, 25, 55] use more information than LIDAR-only ones.”