A Deep Learning-based Radar and Camera Sensor Fusion Architecture for Object Detection

By Felix Nobis, Maximilian Geisslinger, Markus Weber, Johannes Betz, Markus Lienkamp. Sensor Data Fusion, 2019

313 citations recorded by the source corpus.

Object detection in camera images, using deep learning has been proven successfully in recent years. Rising detection rates and computationally efficient network structures are pushing this technique towards application in production vehicles. Nevertheless, the sensor quality of the camera is limited in severe weather conditions and through increased sensor noise in sparsely lit areas and at night. Our approach enhances current 2D object detection networks by fusing camera data and projected sparse radar data in the network layers. The proposed CameraRadarFusion Net (CRF-Net) automatically learns at which level the fusion of the sensor data is most beneficial for the detection result. Additionally, we introduce BlackIn, a training strategy inspired by Dropout, which focuses the learning on a specific sensor type. We show that the fusion network is able to outperform a state-of-the-art image-only network for two different datasets. The code for this research will be made available to the public at: https://github.com/TUMFTM/CameraRadarFusionNet

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/7f06cedeee129ecf7ca1ac382707a19fd85c1ce1

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Sensor and Sensor Fusion Technology in Autonomous Vehicles: A Review (2021, background): “The studies on multi-sensor fusion systems in AVs for environment perception and object detection are well-established in the literature [16,23,92,165,173-175].”
BEVFusion: A Simple and Robust LiDAR-Camera Fusion Framework (2022, background): “There exist other works to leverage other modalities, such as fusing camera-radar by feature map concatenation [3, 18, 31, 30].”
FUTR3D: A Unified Sensor Fusion Framework for 3D Detection (2022, methodology): “Conducting camera-radar fusion usually involves more complicated feature alignment techniques due to the sparsity of radar signals [3, 10, 18, 19].”
FloW: A Dataset and Benchmark for Floating Waste Detection in Inland Waters (2021, methodology): “For the image-radar fusion based object detection, we test two algorithms proposed in [37] and [23].”
Multi-class Road User Detection with 3+1D Radar in the View-of-Delft Dataset (2022, background): “Researchers attempted to overcome these challenges and obtain more information in various ways, e.g.: by merging multiple frames over time [5][22][23], using multiple radars [24], using low-level radar data [3][4][23], or fusing radar with other sensor modalities [25][26][27][28].”
CenterFusion: Center-based Radar and Camera Fusion for 3D Object Detection (2020, background): “CRF-Net [22] also projects radar detections to the image plane, but represents them as vertical lines where the pixel values correspond to the depth of each detection point.”