Deep Multimodal Representation Learning: A Survey

By Wenzhong Guo, Jianwen Wang, Shiping Wang. IEEE Access, 2019

478 citations recorded by the source corpus.

Multimodal representation learning, which aims to narrow the heterogeneity gap among different modalities, plays an indispensable role in the utilization of ubiquitous multimodal data. Due to the powerful representation ability with multiple levels of abstraction, deep learning-based multimodal representation learning has attracted much attention in recent years. In this paper, we provided a comprehensive survey on deep multimodal representation learning which has never been concentrated entirely. To facilitate the discussion on how the heterogeneity gap is narrowed, according to the underlying structures in which different modalities are integrated, we category deep multimodal representation learning methods into three frameworks: joint representation, coordinated representation, and encoder-decoder. Additionally, we review some typical models in this area ranging from conventional models to newly developed technologies. This paper highlights on the key issues of newly developed technologies, such as encoder-decoder model, generative adversarial networks, and attention mechanism in a multimodal representation learning perspective, which, to the best of our knowledge, have never been reviewed previously, even though they have become the major focuses of much contemporary research. For each framework or model, we discuss its basic structure, learning objective, application scenes, key issues, advantages, and disadvantages, such that both novel and experienced researchers can benefit from this survey. Finally, we suggest some important directions for future work.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Signal Processing

View Paper: https://www.semanticscholar.org/paper/c192c7d1d94e7a64de7e18e2f2fdffbf2909fcff

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

On Modality Bias in the TVQA Dataset (2020, unknown): “image, text) has sparked interest in modelling multimodal data, especially over the last five years [1, 3, 12, 16, 22, 36, 50].”
Multimodal Learning With Transformers: A Survey (2022, background): “MML [1], [60], [61] has been an important research area in recent decades; an early multimodal application – audio-visual speech recognition was studied in 1980s [62].”
Recent Advances and Trends in Multimodal Deep Learning: A Review (2021, background): “1 Multimodal Intelligence: Representation Learning, Information Fusion, and Applications [174] IEEE 2020 2 Experience Grounds Language [16] arXiv 2020 3 A Survey on Deep Learning for Multimodal Data Fusion [40] NC(MIT) 2020 4 Trends in integration of vision and language research: A survey of tasks, ”
MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis (2020, background): “shared representation in a common subspace with distributional similarity constraints [18].”
Deep multi-view learning methods: A review (2021, background): “These surveys focus on the problems of specific MVL methods, such as multi-view fusion [10, 27, 28], multi-modal learning [37, 38], multi-view clustering [26] and multi-view representation learning [39, 40].”
Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning (2022, background): “might seem reasonable to attribute the modality gap to differences in data modalities [21], Figure 2”
Multi-document Summarization via Deep Learning Techniques: A Survey (2020, background): “Deep learning is well-suited for multi-modal tasks [58], as it is able to efectively capture highly nonlinear relationships between images, text or video data.”