Recent Advances and Trends in Multimodal Deep Learning: A Review

By Jabeen Summaira, Xi Li, Amin Muhammad Shoib, Songyuan Li, Abdul Jabbar. 2021

71 citations recorded by the source corpus.

Deep Learning has implemented a wide range of applications and has become increasingly popular in recent years. The goal of multimodal deep learning is to create models that can process and link information using various modalities. Despite the extensive development made for unimodal learning, it still cannot cover all the aspects of human learning. Multimodal learning helps to understand and analyze better when various senses are engaged in the processing of information. This paper focuses on multiple types of modalities, i.e., image, video, text, audio, body gestures, facial expressions, and physiological signals. Detailed analysis of past and current baseline approaches and an in-depth study of recent advancements in multimodal deep learning applications has been provided. A fine-grained taxonomy of various multimodal deep learning applications is proposed, elaborating on different applications in more depth. Architectures and datasets used in these applications are also discussed, along with their evaluation metrics. Last, main issues are highlighted separately for each domain along with their possible future research directions.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Ying Shan
2. Qi Tian
3. Fei Wu
4. Yongjian Fu
5. Xi Li
6. Xuewei Li
7. Zhongang Qi
8. Hanbin Zhao

In Computer Vision and Pattern Recognition

1. Ying Shan
2. Xi Li
3. Fei Wu
4. Qi Tian
5. Yongjian Fu
6. Xianpan Zhou
7. Xuewei Li
8. Zhongang Qi

Citations in the corpus

• MgSvF: Multi-Grained Slow versus Fast Framework for Few-Shot Class-Incremental Learning (2020, background): “Incremental learning is usually conducted under the task-incremental [72], [73] or the class-incre-mental learning scenarios [2], [70], [74], [75], [76], [77].”
• LayoutDiffusion: Controllable Diffusion Model for Layout-to-Image Generation (2023, background): “This is mainly due to the ambiguity of the text and its weakness in precisely expressing the position of the image space [6, 15, 22, 43–45].”