Deep Continuous Fusion for Multi-sensor 3D Object Detection

By Ming Liang, Binh Yang, Shenlong Wang, Raquel Urtasun. 2018

938 citations recorded by the source corpus.

In this paper, we propose a novel 3D object detector that can exploit both LIDAR as well as cameras to perform very accurate localization. Towards this goal, we design an end-to-end learnable architecture that exploits continuous convolutions to fuse image and LIDAR feature maps at different levels of resolution. Our proposed continuous fusion layer encode both discrete-state image features as well as continuous geometric information. This enables us to design a novel, reliable and efficient end-to-end learnable 3D object detector based on multiple sensors. Our experimental evaluation on both KITTI as well as a large scale 3D object detection benchmark shows significant improvements over the state of the art.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

nuScenes: A Multimodal Dataset for Autonomous Driving (2019, background): “And while there are several works that have proposed fusion methods based on cameras and lidar [48, 14, 64, 52, 81, 75, 29], PointPillars [51] showed a lidar-only method that performed on par with existing fusion based methods.”
F-cooper: feature based cooperative perception for autonomous vehicle edge computing system using 3D point clouds (2019, background): “For example, [17] fuses features from different convolutional layers in one detection model, [5, 16] fuse features from different sensors within one veihcle.”
PointAugmenting: Cross-Modal Augmentation for 3D Object Detection (2021, methodology): “Researchers [11, 10, 29] have attempted to transform the front-view camera features into BEV maps.”
IntentNet: Learning to Predict Intention from Raw Sensor Data (2018, background): “[19] proposed a real time 3D detector that exploits multiple sensors (i.”
Frustum ConvNet: Sliding Frustums to Aggregate Local Point-Wise Features for Amodal (2019, methodology): “In order to leverage the expertise in 2D object detection, existing methods convert 3D point clouds either into 2D images by view projection [4]–[6], or into regular grids of voxels by quantization [7]–[10].”
What Makes Multimodal Learning Better than Single (Provably) (2021, methodology): “[18] designs a reliable and efficient end-to-end learnable 3D object detector to perform accurate localization by exploiting both LIDAR as well as cameras.”
Fast Point R-CNN (2019, methodology): “Discretizing points into voxelized input [43, 20] or projecting them to BEV with compact shape like RGB images [40, 23] forms a set of solutions, where abstract and rich feature representation can be produced.”
Towards Robust LiDAR-based Perception in Autonomous Driving: General Black-box Adversarial Sensor Attack and Countermeasures (2020, background): “e-art models can be grouped into three classes: 1. Bird’s-eye view (BEV)-based 3D object detection. Due to the remarkable progress made in 2D image recognition tasks, a large number of existing works [7, 40, 43, 68] attempt to transform LiDAR point clouds into the 2D structure for 3D object detectio”
Structure Aware Single-Stage 3D Object Detection From Point Cloud (2020, neutral): “4 ContFuse[11] LiDAR+RGB 94.”
Focal Sparse Convolutional Networks for 3D Object Detection (2022, methodology): “Multi-modal fusion methods [19, 25, 55] use more information than LIDAR-only ones.”
Multimodal Virtual Point 3D Detection (2021, background): “Continuous fusion [20, 29] shares image and Lidar features between their backbones.”
Recent Advancements in End-to-End Autonomous Driving Using Deep Learning: A Survey (2023, background): “Hu et al. [20] employ a cost function that takes advantage of the learned occupancy probability ﬁeld, represented by segmentation maps, and prior knowledge such as trafﬁc rules to select the trajectory with the minimum cost.”
TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers (2022, background): “Similarly, [16, 17, 50, 59] first project a point cloud onto the bird’s eye view (BEV) plane and then fuse the image features with the BEV pixels.”
Safety-Enhanced Autonomous Driving Using Interpretable Sensor Fusion Transformer (2022, background): “Though there are existing works fusing information from multiple sensors, they either match geometric features between image space and LiDAR projection space by locality assumption [3, 4], or simply concatenate multi-sensor features [5, 6].”
Learning Depth-Guided Convolutions for Monocular 3D Object Detection (2019, background): “[29, 28] proposed a continuous fusion layer that encodes both discrete-state image features as well as continuous geometric information.”
OPV2V: An Open Benchmark Dataset and Fusion Pipeline for Perception with Vehicle-to-Vehicle Communication (2021, neutral): “With recent advancements in robotic sensing and machine learning, the reliability of perception has been significantly improved [1], [2], [3], and 3D object detection algorithms have achieved outstanding performance either with LiDAR point clouds [4], [5], [6], [7] or multi-sensor data [8], [9].”
V2X-ViT: Vehicle-to-Everything Cooperative Perception with Vision Transformer (2022, neutral): “With recent advancements of deep learning, the robustness of single-vehicle perception systems has demonstrated significant improvement in several tasks such as semantic segmentation [32,10], depth estimation [54,11], and object detection and tracking [20,23,48,13].”
PointPillars: Fast Encoders for Object Detection From Point Clouds (2018, methodology): “12 N/A N/A N/A N/A N/A N/A Cont-Fuse [15] Lidar & Img.”