Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects

By Dana Lahat, T. Adalı, Christian Jutten. Proceedings of the IEEE, 2015

1,143 citations recorded by the source corpus.

In various disciplines, information about the same phenomenon can be acquired from different types of detectors, at different conditions, in multiple experiments or subjects, among others. We use the term “modality” for each such acquisition framework. Due to the rich characteristics of natural phenomena, it is rare that a single modality provides complete knowledge of the phenomenon of interest. The increasing availability of several modalities reporting on the same system introduces new degrees of freedom, which raise questions beyond those related to exploiting each modality separately. As we argue, many of these questions, or “challenges,” are common to multiple domains. This paper deals with two key issues: “why we need data fusion” and “how we perform it.” The first issue is motivated by numerous examples in science and technology, followed by a mathematical framework that showcases some of the benefits that data fusion provides. In order to address the second issue, “diversity” is introduced as a key concept, and a number of data-driven solutions based on matrix and tensor decompositions are discussed, emphasizing how they account for diversity across the data sets. The aim of this paper is to provide the reader, regardless of his or her community of origin, with a taste of the vastness of the field, the prospects, and the opportunities that it holds.

Subfields: Artificial Intelligence, Media Technology, Signal Processing

View Paper: https://www.semanticscholar.org/paper/2aca01aa1d0a6083986dcd4614b1d0733028dcc2

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Media Technology

In Signal Processing

Citations in the corpus

Semantic Communications: Principles and Challenges (2021, background): “By introducing new degrees of freedom, multimodal data improves performance of the intelligent tasks [71].”
Collaborative Multi-Robot Search and Rescue: Planning, Coordination, Perception, and Active Vision (2020, background): “…parallel data integration with high-level decision making, (b) sequential processing of modalities when different modalities have difference conﬁcende or quality levels, (c) true fusion with high-level features or with multivariate features, and (d) true fusion with minimal reduction [208], [209].”
Generalizing from a Few Examples (2019, methodology): “ in E to make the learning of the supervised targetT feasible. Existing FSL methods use prior knowledge from a single modality, which is the output of a sensor for observing some phenomena or systems [61], such as images, texts, and videos. However, multi-modality prior knowledge can be used to prov”
Cognitive computing for coalition situational understanding (2017, background): “Moreover, the sources will commonly span multiple modalities (for example, imagery, acoustic and textual data [11]) requiring NLP and VSP in addition to ML.”
Comparing Recognition Performance and Robustness of Multimodal Deep Learning Models for Multimodal Emotion Recognition (2021, neutral): “image segmentation, and video classification [36].”
Multimodal machine learning in precision health: A scoping review (2022, background): “There are three main types of data fusion that are used in machine learning; early (data-level), intermediate (joint), and late (decision-level) 30 .”
A Survey on Deep Learning for Multimodal Data Fusion (2020, background): “The reasonable fusion of these multimodal data can help us better understand the event of interest, especially when one modality is incomplete (Khaleghi, Khamis, Karray, & Razavi, 2013; Lahat, Adali, & Jutten, 2015).”
Deep Multimodal Learning: A Survey on Recent Advances and Trends (2017, methodology): “For the purposes of our review, we adopt the definition provided by Lahat et al. [2], where we consider phenomena or systems that are observed using multiple sensors and each sensor output can be termed a modality associated with a single data set.”
A Survey of Multi-View Representation Learning (2016, neutral): “Recently, several related surveys [4], [24], [25], [26], [27] of”
Explainable AI for Intelligence Augmentation in Multi-Domain Operations (2019, background): “Moreover, the sources will commonly span multiple modalities, for example, imagery, acoustic and natural language data (Lahat, Adali, and Jutten 2015).”