A survey on multimodal large language models

By Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, Enhong Chen. National Science Review, 2023

1,509 citations recorded by the source corpus.

ABSTRACT Recently, the multimodal large language model (MLLM) represented by GPT-4V has been a new rising research hotspot, which uses powerful large language models (LLMs) as a brain to perform multimodal tasks. The surprising emergent capabilities of the MLLM, such as writing stories based on images and optical character recognition–free math reasoning, are rare in traditional multimodal methods, suggesting a potential path to artificial general intelligence. To this end, both academia and industry have endeavored to develop MLLMs that can compete with or even outperform GPT-4V, pushing the limit of research at a surprising speed. In this paper, we aim to trace and summarize the recent progress of MLLMs. First, we present the basic formulation of the MLLM and delineate its related concepts, including architecture, training strategy and data, as well as evaluation. Then, we introduce research topics about how MLLMs can be extended to support more granularity, modalities, languages and scenarios. We continue with multimodal hallucination and extended techniques, including multimodal in-context learning, multimodal chain of thought and LLM-aided visual reasoning. To conclude the paper, we discuss existing challenges and point out promising research directions.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/ebedc4d7a2356090904baba4104ef0832bc236df

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models (2023, neutral): “Equipped with the powerful LLM, MLLM is expected to address more complex multi-modal tasks [19, 47, 41].”
Time-LLM: Time Series Forecasting by Reprogramming Large Language Models (2023, background): “Prompting serves as a straightforward yet effective approach t task-specific activation of LLMs (Yin et al., 2023).”
PointLLM: Empowering Large Language Models to Understand Point Clouds (2023, background): “Multi-modal Large Language Models (MLLMs) are designed to comprehend and interpret a wide range of information that extends be-2 yond mere text-based data [51].”
Large language models in healthcare and medical domain: A review (2023, background): “This theoretical introduction highlights the transformative potential of MLLMs and their central role in pushing the boundaries of artificial intelligence, affecting areas such as image and speech recognition, content generation, and interactive AI applications [30].”
A Survey on Multimodal Large Language Models for Autonomous Driving (2023, background): “The modality alignment enables them to execute a diverse array of tasks with higher proficiency, including categorizing images, matching text to corresponding videos, and speech detection.”
Hallusionbench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models (2023, background): “Most recently, researchers have been creating high-quality, diverse multi-modal datasets from GPT4 and GPT-4V [48] to fine-tune open-source LVLMs, including LLaVA [33], MiniGPT4 [63], Mplug-Owl [50], LRV-Instruction [29], LLaVAR [60] and other works [12, 25, 37, 52].”
VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use (2023, neutral): “As a result, it is difficult to assess progress in this direction, which both reduces the field’s ability to identify true breakthroughs and increases vulnerability to potential pitfalls of evaluation that have hampered progress in other areas of machine learning [68, 81].”
Visual Instruction Tuning with Polite Flamingo (2023, background): “Research on enabling visual perception for powerful but blind LLMs attracted widespread attention recently (Yin et al. 2023a).”
MoE-LLaVA: Mixture of Experts for Large Vision-Language Models (2024, background): “However, the cost of scaling up dense visual data and models is challenging to bear (Liu et al., 2022; Yin et al., 2023).”
Large language models for generative information extraction: a survey (2023, neutral): “We briefly describe some backbones that are commonly used in the field of generative information extraction, which is shown in Table 8.”
Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis (2024, neutral): “The rapid development of MLLMs in recent years [66] has highlighted their impressive perception and cognitive capabilities across various multimodal benchmarks [16, 43, 67].”
Show-o: One Single Transformer to Unify Multimodal Understanding and Generation (2024, background): “2.1 M ULTIMODAL U NDERSTANDING Significant advancements in large language models (LLMs) (Touvron et al., 2023; Brown et al., 2020; Chowdhery et al., 2023) have inspired the development of multimodal large language models (MLLMs) (Li et al., 2024; Yin et al., 2023; Bai et al., 2024).”
A Survey on Hallucination in Large Vision-Language Models (2024, background): “However, the restricted quantity of these tokens prevents encoding all the information present in images during the alignment process [ Yin et al. , 2023a; Chen et al. , 2023c ] , and consequent information loss raises the risk of hallucinations.”
LLaVA-Prumerge: Adaptive Token Reduction for Efficient Large Multimodal Models (2024, positive): “Large Multimodal Models (LMMs) [30, 48, 52, 55] extend these reasoning capabilities to images, where given an image and an associated question, a vision encoder and an LLM are leveraged to generate text responses in a chat format.”
A Survey on the Memory Mechanism of Large Language Model-based Agents (2024, neutral): “In addition, cognitive psychology has been studied for a long time, so many effective human memory theories and architectures have been accumulated, which can support more advanced capabilities of the agents [147].”
LLaVA-OneVision: Easy Visual Task Transfer (2024, positive): “Specifically, we adopt the perception benchmarks including MME [150], MMBench [86], and MMVet [153], and reasoning benchmarks such as MathVerse [164], MathVista [90], and MMMU [156].”
MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark (2024, background): “Recent advances in multimodal large language models (MLLMs) have led to remarkable progress in tackling complex reasoning tasks that combine textual and visual information (Yin et al., 2023a; Jin et al., 2024).”
A Comprehensive Overview of Large Language Models (2023, neutral): “Different from previous works [230], [231] that rely on limited VQA datasets and small-scale neural networks, current LLMaided methods offer benefits of stronger generalization ability, emergent ability, and interactivity [214].”
InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Composition (2023, background): “It consistently achieves state-of-the-art performance across various leading benchmarks for visionlanguage large models, encompassing MME Benchmark [18, 71], MMBench [72], Seed-Bench [29] in English, and MMBench-CN [72], CCBench (Chinese Cultural Benchmark) [72] for evaluations in Chinese.”