EPNet: Enhancing Point Features with Image Semantics for 3D Object Detection

By Tengteng Huang, Zhe Liu, Xiwu Chen, Xiang Bai. European Conference on Computer Vision, 2020

490 citations recorded by the source corpus.

In this paper, we aim at addressing two critical issues in the 3D detection task, including the exploitation of multiple sensors~(namely LiDAR point cloud and camera image), as well as the inconsistency between the localization and classification confidence. To this end, we propose a novel fusion module to enhance the point features with semantic image features in a point-wise manner without any image annotations. Besides, a consistency enforcing loss is employed to explicitly encourage the consistency of both the localization and classification confidence. We design an end-to-end learnable framework named EPNet to integrate these two components. Extensive experiments on the KITTI and SUN-RGBD datasets demonstrate the superiority of EPNet over the state-of-the-art methods. Codes and models are available at: \url{this https URL}.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Geology

View Paper: https://www.semanticscholar.org/paper/ed77a2e787aa06308072f78c8b06265c0fa23d9e

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In Geology

Citations in the corpus

3D Object Detection for Autonomous Driving: A Comprehensive Survey (2022, background): “In addition to the fusion in grid-based backbones, there also exist some papers incorporating image information into the point-based detection backbones [327, 109, 324, 308, 387].”
Unifying Voxel-based Representation with Transformer for 3D Object Detection (2022, background): “Specifically, point-level fusion [30, 31, 32, 7] combines features from different modalities at the early stage of the network, which enables sufficient interaction.”
Multimodal Virtual Point 3D Detection (2021, background): “Continuous fusion [20, 29] shares image and Lidar features between their backbones.”
DeepFusion: Lidar-Camera Deep Fusion for Multi-Modal 3D Object Detection (2022, methodology): “the features are combined after feature extraction [5,14,18].”
PointAugmenting: Cross-Modal Augmentation for 3D Object Detection (2021, methodology): “Instead, other methods [7, 16, 19] explore the fusion mechanism in a point-wise manner.”
BEVFusion: A Simple and Robust LiDAR-Camera Fusion Framework (2022, methodology): “Previous fusion methods can be broadly categorized into (a) point-level fusion mechanism [43, 46, 47, 46, 17, 60] that project image features onto raw point clouds, and (b) feature-level fusion mechanism [4, 61, 1, 21] that projects LiDAR feature or proposals on each view image separately to…”
Focal Sparse Convolutional Networks for 3D Object Detection (2022, methodology): “Multi-modal fusion methods [19, 25, 55] use more information than LIDAR-only ones.”
TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers (2022, background): “They first find a hard association between LiDAR points and image pixels based on calibration matrices, and then augment LiDAR features with the segmentation scores [47,52] or CNN features [10, 22, 40, 48, 63] of the associated pixels through point-wise concatenation.”