Deep Multi-Modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges

By Di Feng, Christian Haase-Schuetz, Lars Rosenbaum, Heinz Hertlein, Fabian Duffhauss, Claudius Gläser, Werner Wiesbeck, Klaus C. J. Dietmayer. 2019

1,343 citations recorded by the source corpus.

Recent advancements in perception for autonomous driving are driven by deep learning. In order to achieve robust and accurate scene understanding, autonomous vehicles are usually equipped with different sensors (e.g. cameras, LiDARs, Radars), and multiple sensing modalities can be fused to exploit their complementary properties. In this context, many methods have been proposed for deep multi-modal perception problems. However, there is no general guideline for network architecture design, and questions of “what to fuse”, “when to fuse”, and “how to fuse” remain open. This review paper attempts to systematically summarize methodologies and discuss challenges for deep multi-modal object detection and semantic segmentation in autonomous driving. To this end, we first provide an overview of on-board sensors on test vehicles, open datasets, and background information for object detection and semantic segmentation in autonomous driving research. We then summarize the fusion methodologies and discuss challenges and open questions. In the appendix, we provide tables that summarize topics and methods. We also provide an interactive online platform to navigate each reference: https://boschresearch.github.io/multimodalperception/.

Subfields: Automotive Engineering, Computer Vision and Pattern Recognition, Media Technology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Automotive Engineering

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

A Review and Comparative Study on Probabilistic Object Detection in Autonomous Driving (2020, background): “Probabilistic object detectors usually adapt base networks from well-studied deterministic object detectors to capture epistemic uncertainty using Deep Ensembles or MC-Dropout [8], [80]–[83].”
Milestones in Autonomous Driving and Intelligent Vehicles: Survey of Surveys (2023, neutral): “[45], [46], [47], [48], [49], [50], [51], [52], [53], [54], [55], [56], [57], [58], [59], [60], [61], [62], [63], [64], [65], [66], [67], scene understanding [68], [69], [70], tracking [71], [72],”
Cooperative Perception for 3D Object Detection in Driving Scenarios Using Infrastructure Sensors (2019, background): “A dedicated comprehensive review of 3D object detection techniques, covering all categories, can be found in [5], [13].”
Multimodal End-to-End Autonomous Driving (2019, background): “For more details on multimodal object detection, the reader is referred to recent surveys [100], [97].”
Opportunities and Challenges in Explainable Artificial Intelligence (XAI): A Survey (2020, background): “in healthcare [1]–[3], ophthalmology [4]–[6], developmental disorders [7]–[9], in autonomous robots and vehicles [10]–[12], in image processing classification and detection [13], [14], in speech and audio processing [15], [16], cyber-security [17], [18], and many more indicate the reach of DL algori”
Deep Learning Sensor Fusion for Autonomous Vehicle Perception and Localization: A Review (2020, neutral): “Based on the source of the fused data, sensor fusion can also be categorized as a multimodal method [61], where the fused data are obtained from two or more di ﬀ erent types of Another class of this category is multitemporal fusion, where data are obtained from the same sensor but at different…”
Drone-Based RGB-Infrared Cross-Modality Vehicle Detection Via Uncertainty-Aware Learning (2020, positive): “These RGB-Infrared cross-modality images, while introducing effective information, also introduce redundant information [20]–[22].”
CenterFusion: Center-based Radar and Camera Fusion for 3D Object Detection (2020, background): “Early fusion methods are usually sensitive to spatial or temporal misalignment of the data [8].”
ProcTHOR: Large-Scale Embodied AI Using Procedural Generation (2022, background): “In the broader field of computer vision, the dataset may be used to study object detection [64]; NeRFs [80, 110, 43, 71]; segmentation, depth, and optimal flow estimation [35, 43]; generative modeling [59, 62, 61]; occlusion reasoning [34]; and pose estimation [19], among others.”
Behavioral Use Licensing for Responsible AI (2020, background): “, face recognition), detection of cancerous regions in a mammogram [57], or for classifying buildings and people from an unmanned vehicle [27].”
A Multilevel Multimodal Fusion Transformer for Remote Sensing Semantic Segmentation (2024, background): “In contrast, middle fusion can capture cross-modal dependencies of feature representations, making it more effective in the context of representation learning [37].”
PIDNet: A Real-time Semantic Segmentation Network Inspired by PID Controllers (2022, background): “Even though these models achieve encouraging segmentation accuracy, too much computational cost are required, which significantly hinder their application in real-time scenarios, such as autonomous vehicle [16] and robot surgery [44].”
Multimodal Learning With Transformers: A Survey (2022, background): “For instance, an AI navigation robot needs multimodal sensors to perceive the real-world environment [66], [67], [68], e.”
A Deep Learning-based Radar and Camera Sensor Fusion Architecture for Object Detection (2019, background): “[26] gives an overview of deep learning methods for sensor fusion.”
PolyLaneNet: Lane Estimation via Deep Polynomial Regression (2020, background): “The perceptual problems related to this field have been immensely impacted by the advances in deep learning [2]–[4].”
A Survey of End-to-End Driving: Architectures and Training Methods (2020, background): “More specialized reviews and comparisons of methods exist on this topic, in studies such as [102], [103].”