Multimodal Large Language Models: A Survey

By Jiayang Wu, Wensheng Gan, Zefeng Chen, Shicheng Wan, Philip S. Yu. BigData Congress [Services Society], 2023

406 citations recorded by the source corpus.

The exploration of multimodal language models integrates multiple data types, such as images, text, language, audio, and other heterogeneity. While the latest large language models excel in text-based tasks, they often struggle to understand and process other data types. Multimodal models address this limitation by combining various modalities, enabling a more comprehensive understanding of diverse data. This paper begins by defining the concept of multimodal and examining the historical development of multimodal algorithms. Furthermore, we introduce a range of multimodal products, focusing on the efforts of major technology companies. A practical guide is provided, offering insights into the technical aspects of multimodal models. Moreover, we present a compilation of the latest algorithms and commonly used datasets, providing researchers with valuable resources for experimentation and evaluation. Lastly, we explore the applications of multimodal models and discuss the challenges associated with their development. By addressing these aspects, this paper aims to facilitate a deeper understanding of multimodal models and their potentiality in various domains.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/52941cadbd340344f3e0a6f50719fe55b3de5088

Citations in the corpus