Multimodal Learning With Transformers: A Survey

By P. Xu, Xiatian Zhu, David A. Clifton. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022

1,083 citations recorded by the source corpus.

Transformer is a promising neural network learner, and has achieved great success in various machine learning tasks. Thanks to the recent prevalence of multimodal applications and Big Data, Transformer-based multimodal learning has become a hot topic in AI research. This paper presents a comprehensive survey of Transformer techniques oriented at multimodal data. The main contents of this survey include: (1) a background of multimodal learning, Transformer ecosystem, and the multimodal Big Data era, (2) a systematic review of Vanilla Transformer, Vision Transformer, and multimodal Transformers, from a geometrically topological perspective, (3) a review of multimodal Transformer applications, via two important paradigms, i.e., for multimodal pretraining and for specific multimodal tasks, (4) a summary of the common challenges and designs shared by the multimodal Transformer models and applications, and (5) a discussion of open problems and potential research directions for the community.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/622428f5122ad12a40229e1768ecb929fd747ee7

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Transformer-Based Visual Segmentation: A Survey (2023, methodology): “• Contribution: In this survey, we systematically introduce recent advances in transformer-based visual segmentation meth-ods.”
Large-scale Multi-modal Pre-trained Models: A Comprehensive Survey (2023, background): “27 Multimodal Learning with Transformers: A Survey [50] 2022 arXiv MM DC, 23”
Vision-Language Models for Vision Tasks: A Survey (2023, background): “Xu et al. [53] and Wang et al. [54] shared recent progress of multi-modal learning on multi-modal tasks.”
CTP: Towards Vision-Language Continual Pretraining via Compatible Momentum Contrast and Topology Preservation (2023, background): “However, the loose modal interaction by cosine similarity also limits the multi-modal fusion ability [4, 10, 70].”
Foundation Models Defining a New Era in Vision: A Survey and Outlook (2025, neutral): “The work of [26] reviews transformer techniques for multimodal data with a survey of vanilla transformers, vision transformers, and multimodal transformers from a geometrically topological perspective.”
A Survey of Knowledge Graph Reasoning on Graph Types: Static, Dynamic, and Multi-Modal (2022, background): “However, most existing artificial intelligence (AI) models only imitate the learning procedure from experiences while ignoring the former [1], [2], [3], [4], thus making them less explainable and worse performances.”