MM-LLMs: Recent Advances in MultiModal Large Language Models

By Duzhen Zhang, Yahan Yu, Chenxing Li, Jiahua Dong, Dan Su, Chenhui Chu, Dong Yu. Annual Meeting of the Association for Computational Linguistics, 2024

448 citations recorded by the source corpus.

In the past year, MultiModal Large Language Models (MM-LLMs) have undergone substantial advancements, augmenting off-the-shelf LLMs to support MM inputs or outputs via cost-effective training strategies. The resulting models not only preserve the inherent reasoning and decision-making capabilities of LLMs but also empower a diverse range of MM tasks. In this paper, we provide a comprehensive survey aimed at facilitating further research of MM-LLMs. Initially, we outline general design formulations for model architecture and training pipeline. Subsequently, we introduce a taxonomy encompassing 126 MM-LLMs, each characterized by its specific formulations. Furthermore, we review the performance of selected MM-LLMs on mainstream benchmarks and summarize key training recipes to enhance the potency of MM-LLMs. Finally, we explore promising directions for MM-LLMs while concurrently maintaining a real-time tracking website for the latest developments in the field. We hope that this survey contributes to the ongoing advancement of the MM-LLMs domain.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/a050c9b0c321839e4427ab9defa3463be7825ac4

Citations in the corpus

• LLaVA-Prumerge: Adaptive Token Reduction for Efficient Large Multimodal Models (2024, unknown): “Vision Transformers (ViTs) [12] are the most widely used vision encoder for LMMs, in which the input image is converted into a sequence of representative tokens by the ViT, and then fed into an LLM for understanding [18, 29, 52, 55].”
• Do Agents Dream of False Memories? Black-box Visual Attacks on Long-term Memory in Multimodal AI Agents (2026, unknown): “For image-bearing turns, such as a user uploading an image of a meal or a product, the image is encoded through a visual encoder and stored alongside associated text [81].”