Large-scale Multi-modal Pre-trained Models: A Comprehensive Survey

By Xiao Wang, Guangyao Chen, Guangwu Qian, Pengcheng Gao, Xiaoyong Wei, Yaowei Wang, Yonghong Tian, Wen Gao. Machine Intelligence Research, 2023

326 citations recorded by the source corpus.

With the urgent demand for generalized deep models, many pre-trained big models are proposed, such as bidirectional encoder representations (BERT), vision transformer (ViT), generative pre-trained transformers (GPT), etc. Inspired by the success of these models in single domains (like computer vision and natural language processing), the multi-modal pre-trained big models have also drawn more and more attention in recent years. In this work, we give a comprehensive survey of these models and hope this paper could provide new insights and helps fresh researchers to track the most cutting-edge works. Specifically, we firstly introduce the background of multi-modal pre-training by reviewing the conventional deep learning, pre-training works in natural language process, computer vision, and speech. Then, we introduce the task definition, key challenges, and advantages of multi-modal pre-training models (MM-PTMs), and discuss the MM-PTMs with a focus on data, objectives, network architectures, and knowledge enhanced pre-training. After that, we introduce the downstream tasks used for the validation of large-scale MM-PTMs, including generative, classification, and regression tasks. We also give visualization and analysis of the model parameters and results on representative downstream tasks. Finally, we point out possible research directions for this topic that may benefit future works. In addition, we maintain a continuously updated paper list for large-scale pre-trained multi-modal big models: https://github.com/wangxiao5791509/MultiModal_BigModels_Survey.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/746bb45433f6b24d3ae64d6cd51c4e9d00a0ffa7

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Vision-Language Models for Vision Tasks: A Survey (2023, background): “Xu et al. [53] and Wang et al. [54] shared recent progress of multi-modal learning on multi-modal tasks.”
A Survey on Vision--Language--Action Models for Embodied AI (2024, unknown): “Human behavior offers plentiful guidance for robot policies due to its dexterity and diversity.”
Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis (2024, unknown): “Advancements in MLLMs. Recent advancements in MLLMs have seen notable progress [7, 19, 22, 59, 66].”
Large Selective Kernel Network for Remote Sensing Object Detection (2023, methodology): “Transformer-based [59] models, such as the Vision Transformer (ViT) [1,11,14,53,60], Swin transformer [25, 39, 51, 64, 70, 83], and pyramid transformer [62, 67], have gained popularity in computer vision.”
Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Driving (2023, background): “Pioneering work in Visual Language Models (VLMs) has begun to bridge this gap by merging visual and text modalities [9], enabling spatial reasoning with the power of pre-trained LLMs.”
A Survey on Graph Neural Networks for Time Series: Forecasting, Classification, Imputation, and Anomaly Detection (2023, background): “The exploration of pre-training strategies and GNN transferability for time series tasks is a burgeoning research area, especially in the current era of generative AI and large models, which showcase the potential for a single, multimodal model to address diverse tasks [291].”
Multimodal Large Language Models: A Survey (2023, unknown): “Wang et al. [8] presented a comprehensive compilation of the latest algorithms used in multimodal large-scale models and the datasets employed in recent experiments, offering convenience to readers.”
Segment Anything Model for Medical Image Segmentation: Current Applications and Future Directions (2024, background): “The emergence of large-scale foundation models [6, 7] has revolutionized artificial intelligence and sparked a new era due to their remarkable zero-shot and few-shot generalization abilities across a wide range of downstream tasks [8].”