A Survey on Deep Learning for Multimodal Data Fusion

By Jing Gao, Peng Li, Zhikui Chen, Jianing Zhang. Neural Computation, 2020

733 citations recorded by the source corpus.

With the wide deployments of heterogeneous networks, huge amounts of data with characteristics of high volume, high variety, high velocity, and high veracity are generated. These data, referred to multimodal big data, contain abundant intermodality and cross-modality information and pose vast challenges on traditional data fusion methods. In this review, we present some pioneering deep learning models to fuse these multimodal big data. With the increasing exploration of the multimodal big data, there are still some challenges to be addressed. Thus, this review presents a survey on deep learning for multimodal data fusion to provide readers, regardless of their original community, with the fundamentals of multimodal deep learning fusion method and to motivate new multimodal data fusion techniques of deep learning. Specifically, representative architectures that are widely used are summarized as fundamental to the understanding of multimodal deep learning. Then the current pioneering multimodal data fusion deep learning models are summarized. Finally, some challenges and future topics of multimodal data fusion deep learning models are described.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Media Technology

View Paper: https://www.semanticscholar.org/paper/4c7c70760029e813ef76ea5a578174d2d3ec1490

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

Deep Multimodal Data Fusion (2024, background): “Compared to the existing surveys [2, 4, 45, 46, 121, 128, 227, 235, 243] that usually focus on one single task (such as multimodal object recognition) with one specific combination of two modalities (such as RGB+depth data), this survey owns a broader scope covering various modalities and their…”
Deep learning in cancer diagnosis, prognosis and treatment selection (2021, background): “CNNs, RNNs, deep belief networks and autoencoders (AE) [21]; score-level fusion [22]; or multimodal data fusion [23].”
Recent Advances and Trends in Multimodal Deep Learning: A Review (2021, methodology): “The fundamentals of MMDL fusion methods and their challenges are discussed by Gao et al. [40].”
Mixture-of-Experts in Remote Sensing: A Survey (2026, unknown): “As a result, they often struggle to generalize across sensors, domains and tasks without extensive re-training or careful domain adaptation [22, 34].”