Multimodal Foundation Models: From Specialists to General-Purpose Assistants

By Chunyuan Li, Zhe Gan, Zhengyuan Yang, Jianwei Yang, Linjie Li, Lijuan Wang, Jianfeng Gao. 2023

394 citations recorded by the source corpus.

This paper presents a comprehensive survey of the taxonomy and evolution of multimodal foundation models that demonstrate vision and vision-language capabilities, focusing on the transition from specialist models to general-purpose assistants. The research landscape encompasses five core topics, categorized into two classes. (i) We start with a survey of well-established research areas: multimodal foundation models pre-trained for specific purposes, including two topics -- methods of learning vision backbones for visual understanding and text-to-image generation. (ii) Then, we present recent advances in exploratory, open research areas: multimodal foundation models that aim to play the role of general-purpose assistants, including three topics -- unified vision models inspired by large language models (LLMs), end-to-end training of multimodal LLMs, and chaining multimodal tools with LLMs. The target audiences of the paper are researchers, graduate students, and professionals in computer vision and vision-language multimodal communities who are eager to learn the basics and recent advances in multimodal foundation models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

In Multimodal Learning

Citations in the corpus

Visual Instruction Tuning (2023, background): “One of the core aspirations in artificial intelligence is to develop a general-purpose assistant that can effectively follow multi-modal vision-and-language instructions, aligned with human intent to complete various real-world tasks in the wild [4, 27, 26].”
Improved Baselines with Visual Instruction Tuning (2023, background): “Large multimodal models (LMMs) have become increasingly popular in the research community, as they are the key building blocks towards general-purpose assistants [2, 29, 42].”
Show-o: One Single Transformer to Unify Multimodal Understanding and Generation (2024, background): “2.1 M ULTIMODAL U NDERSTANDING Significant advancements in large language models (LLMs) (Touvron et al., 2023; Brown et al., 2020; Chowdhery et al., 2023) have inspired the development of multimodal large language models (MLLMs) (Li et al., 2024; Yin et al., 2023; Bai et al., 2024).”
Cognitive Architectures for Language Agents (2023, background): “We suggest the boundary question can be answered in terms of controllability and coupling .”
Generative Multimodal Models are In-Context Learners (2023, background): “Multimodal tasks [25, 41] encompass anything involving understanding and generation in single or multiple modalities [5, 19, 58], which can be highly diverse and long-tail.”
Video Instruction Tuning With Synthetic Data (2024, neutral): “We are in an era where large-scale computing and data is crucial for multimodal learning (Li et al., 2024d).”
An Empirical Study of Scaling Instruct-Tuned Large Multimodal Models (2023, background): “We ﬁrst compare our large checkpoints on two recent benchmarks which are speciﬁcally designed for LMM, then report our ﬁndings in the course of scaling up LLaVA models.”
LLaVA-OneVision: Easy Visual Task Transfer (2024, neutral): “It is a core aspiration in AI to build general-purpose assistants with Large Multimodal Models (LMM) [67].”
LLaVA-Video: Video Instruction Tuning With Synthetic Data (2024, neutral): “We are in an era where large-scale computing and data is crucial for multimodal learning (Li et al., 2024d).”
MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts (2023, background): “…(Methani et al., 2020; Masry et al., 2022; Kahou et al., 2017; Chang et al., 2022; Kafle et al., 2018), documents (text in images) (Singh et al., 2019; Mathew et al., 2022; Liu et al., 2023c), or synthetic images (Dahlgren Lindström & Abraham, 2022; Li et al., 2023; Bitton-Guetta et al., 2023).”
LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day (2023, background): “…inputs, the resulting large multimodal models (LMMs) exhibit strong zero-shot task completion performance on a variety of user-oriented vision-language tasks such as image understanding and reasoning, paving the way to develop general-purpose multimodal conversational assistants [22, 2, 23, 10].”
An Embodied Generalist Agent in 3D World (2023, positive): “…et al., 2021) emerge as a promising paradigm in building such generalist models in natural language processing (OpenAI, 2022; 2023), computer vision (Kirillov et al., 2023; Alayrac et al., 2022), and robotics (Brohan et al., 2022; 2023; Reed et al., 2022; Driess et al., 2023; Li et al., 2023c).”