Multimodal Intelligence: Representation Learning, Information Fusion, and Applications

By Chao Zhang, Zichao Yang, Xiaodong He, Li Deng. IEEE Journal on Selected Topics in Signal Processing, 2019

469 citations recorded by the source corpus.

Deep learning methods haverevolutionized speech recognition, image recognition, and natural language processing since 2010. Each of these tasks involves a single modality in their input signals. However, many applications in the artificial intelligence field involve multiple modalities. Therefore, it is of broad interest to study the more difficult and complex problem of modeling and learning across multiple modalities. In this paper, we provide a technical review of available models and learning methods for multimodal intelligence. The main focus of this review is the combination of vision and natural language modalities, which has become an important topic in both the computer vision and natural language processing research communities. This review provides a comprehensive analysis of recent works on multimodal deep learning from three perspectives: learning multimodal representations, fusing multimodal signals at various levels, and multimodal applications. Regarding multimodal representation learning, we review the key concepts of embedding, which unify multimodal signals into a single vector space and thereby enable cross-modality signal processing. We also review the properties of many types of embeddings that are constructed and learned for general downstream tasks. Regarding multimodal fusion, this review focuses on special architectures for the integration of representations of unimodal signals for a particular task. Regarding applications, selected areas of a broad interest in the current literature are covered, including image-to-text caption generation, text-to-image generation, and visual question answering. We believe that this review will facilitate future studies in the emerging field of multimodal intelligence for related communities.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/415efb7b4d9d1e5b64dbaf3fe4229ad462acce71

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

On Modality Bias in the TVQA Dataset (2020, unknown): “The visual question answering (VQA) [1, 23] task in particular has inspired a wealth of multimodal fusion techniques and models [51], and taxonomic analysis of the latent representations they produce [22].”
Multimodal Learning With Transformers: A Survey (2022, background): “, image classification) – usually regarded as a single modality learning application in conventional MML surveys [1], [11], [12] – as a special MML application.”
Recent Advances and Trends in Multimodal Deep Learning: A Review (2021, background): “1 Multimodal Intelligence: Representation Learning, Information Fusion, and Applications [174] IEEE 2020 2 Experience Grounds Language [16] arXiv 2020 3 A Survey on Deep Learning for Multimodal Data Fusion [40] NC(MIT) 2020 4 Trends in integration of vision and language research: A survey of tasks, ”