mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality

By Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yi Zhou, Junyan Wang, Anwen Hu, Pengcheng Shi, Yaya Shi and 7 more. 2023

1,270 citations recorded by the source corpus.

Large language models (LLMs) have demonstrated impressive zero-shot abilities on a variety of open-ended tasks, while recent research has also explored the use of LLMs for multi-modal generation. In this study, we introduce mPLUG-Owl, a novel training paradigm that equips LLMs with multi-modal abilities through modularized learning of foundation LLM, a visual knowledge module, and a visual abstractor module. This approach can support multiple modalities and facilitate diverse unimodal and multimodal abilities through modality collaboration. The training paradigm of mPLUG-Owl involves a two-stage method for aligning image and text, which learns visual knowledge with the assistance of LLM while maintaining and even improving the generation abilities of LLM. In the first stage, the visual knowledge module and abstractor module are trained with a frozen LLM module to align the image and text. In the second stage, language-only and multi-modal supervised datasets are used to jointly fine-tune a low-rank adaption (LoRA) module on LLM and the abstractor module by freezing the visual knowledge module. We carefully build a visually-related instruction evaluation set OwlEval. Experimental results show that our model outperforms existing multi-modal models, demonstrating mPLUG-Owl's impressive instruction and visual understanding ability, multi-turn conversation ability, and knowledge reasoning ability. Besides, we observe some unexpected and exciting abilities such as multi-image correlation and scene text understanding, which makes it possible to leverage it for harder real scenarios, such as vision-only document comprehension. Our code, pre-trained model, instruction-tuned models, and evaluation set are available at https://github.com/X-PLUG/mPLUG-Owl. The online demo is available at https://www.modelscope.cn/studios/damo/mPLUG-Owl.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

In Multimodal Learning

Citations in the corpus

The False Promise of Imitating Proprietary LLMs (2023, methodology): “For example, the training process of many recent vision-language model (Li et al., 2022; Liu et al., 2023; Ye et al., 2023; Zhu et al., 2023) includes ChatGPT or GPT-4 outputs at some stages.”
Improved Baselines with Visual Instruction Tuning (2023, background): “Recent works further demonstrate improved performance by scaling up the pretraining data [3, 13, 53], instruction-following data [13, 17, 28, 57], visual encoders [3], or language models [38], respectively.”
LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention (2023, background): “The current working directory is /home/user/ echo “I am a chatbot“ > file.txt && cat file.txt && ls -l I am a chatbot -rw-r–r– 1 user user”
Hallucination of Multimodal Large Language Models: A Survey (2024, positive): “MLLMs [22, 75, 111, 138] typically refers to a series of models that enable LLMs to perceive and comprehend data from various modalities.”
MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models (2023, neutral): “Existing methods tend to use GPT or manual scoring [28, 33, 50], but there may be problems of inaccuracy and subjectivity.”
EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding (2023, background): “[60] proposes a training strategy to add image & video modality to pretrained large language models.”
A survey on multimodal large language models (2023, positive): “Hybrid Composition Apart from the M-IT data, language-only user-assistant conversation data can also be used to improve conversational proficiencies and instruction-following abilities [22,31,32,35].”
EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought (2023, background): “VideoChat[38], mPLUG-Owl [39] and X-LLM [40], further expand support for video input.”
Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception (2024, background): “As Multimodal Large Language Models (MLLM) Liu et al. [2023d], Zhu et al. [2023], Ye et al. [2023a], Dai et al. [2023], Liu et al. [2023e], Chen et al. [2023], Ye et al. [2023b], Bai et al. [2023], Lin et al. [2023] rapidly progress and exhibit remarkably visual comprehension capabilities, the…”
SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models (2023, methodology): “Different from existing task-specific MLLM models (Ye et al., 2023; Peng et al., 2023b; Chen et al., 2023b; Liu et al., 2023d; Gao et al., 2023b), we integrate a diverse set of visual instruction tasks to tune the pre-trained model, aiming to acquire a wide range of capabilities.”
MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems? (2024, background): “The subsequent mPLUG-Owl [61, 62], Qwen-VL [3], InternLM-XComposer [16], and SPHINX [36, 21] further push the frontier of MLLMs in understanding and generalizing across visual contexts.”
Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond (2023, neutral): “…vision-language models (LVLMs) based on LLMs (Alayrac et al., 2022; Chen et al., 2022; Li et al., 2023c; Dai et al., 2023; Huang et al., 2023; Peng et al., 2023; Zhu et al., 2023; Liu et al., 2023; Ye et al., 2023b,a; Chen et al., 2023a; Li et al., 2023a; Zhang et al., 2023; Sun et al., 2023).”
Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding (2023, methodology): “, 2023), mPLUG-owl (Ye et al., 2023) and MiniGPT4 (Zhu et al.”
MMBench: Is Your Multi-modal Model an All-around Player? (2023, methodology): “During the instruction tuning, Low-Rank Adaptation (LoRA [22]) has been adopted by recent works [49, 13, 11] on language models to achieve better performance on multimodal understanding.”
MovieChat: From Dense Token to Sparse Memory for Long Video Understanding (2023, background): “Existing vision-centric MLLMs follow the paradigm that utilizing pre-trained LLMs and visual encoder with additional learnable modules (Q-former [21, 34, 36, 82] or simple projection layer [22, 39, 43, 61]).”
Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic (2023, background): “Mini-GPT4 (Zhu et al., 2023), mPLUG-OWL (Ye et al., 2023), VPGTrans (Zhang et al., 2023a), and InstructBLIP (Dai et al., 2023) retain Q-Former, replace language model to a larger one, and then tuning on meticulously collected instruction data.”
Are We on the Right Way for Evaluating Large Vision-Language Models? (2024, methodology): “Encouraged by the rapid development of large language models (LLMs) [47, 4, 8, 9, 13, 1, 43], integrating visual modality into LLMs to enhance models’ interactivity capabilities has witnessed ever-changing advances in recent days [54, 26, 24, 11, 52, 2, 48, 31, 5, 12].”
MVBench: A Comprehensive Multi-modal Video Understanding Benchmark (2023, neutral): “Our results indicate that VideoChat2 holds its own against current SOTA methods [77, 94, 96] on these complex reasoning tasks, which underscores the effectiveness and robustness of VideoChat2, especially for long videos.”