mPLUG-OwI2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration

By Qinghao Ye, Haiyang Xu, Jiabo Ye, Mingshi Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, Jingren Zhou. Computer Vision and Pattern Recognition, 2023

705 citations recorded by the source corpus.

Multi-modal Large Language Models (MLLMs) have demonstrated impressive instruction abilities across various open-ended tasks. However, previous methods primarily fo-cus on enhancing multi-modal capabilities. In this work, we introduce a versatile multi-modal large language model, mPLUG-Owl2, which effectively leverages modality collab-oration to improve performance in both text and multi-modal tasks. mPLUG-Owl2 utilizes a modularized network design, with the language decoder acting as a universal interface for managing different modalities. Specifically, mPLUG-Owl2 incorporates shared functional modules to facilitate modal-ity collaboration and introduces a modality-adaptive module that preserves modality-specific features. Extensive experi-ments reveal that mPLUG-Owl2 is capable of generalizing both text tasks and multi-modal tasks and achieving state-of-the-art performances with a single generic model. Notably, mPLUG-Owl2 is the first MLLM model that demonstrates the modality collaboration phenomenon in both pure-text and multi-modal scenarios, setting a pioneering path in the development of future multi-modal foundation models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/ad13b213681b6f634bc83a264df246e83dd9a9d9

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

SVIT: Scaling up Visual Instruction Tuning (2023, neutral): “5 to state-of-the-art MLLMs: BLIP-2 (Li et al., 2023c), InstructBLIP (Dai et al., 2023), Shikra (Chen et al., 2023), IDEFICS (IDEFICS, 2023), Qwen-VL(-Chat) (Bai et al., 2023), mPLUG-Owl2 (Ye et al., 2023) and LLaVA-v1.”
OCRBench: on the hidden mystery of OCR in large multimodal models (2023, neutral): “Additionally, [22, 23] emphasize modality collaboration for image and text.”
MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems? (2024, background): “The subsequent mPLUG-Owl [61, 62], Qwen-VL [3], InternLM-XComposer [16], and SPHINX [36, 21] further push the frontier of MLLMs in understanding and generalizing across visual contexts.”
Q-Align: Teaching LMMs for Visual Scoring via Discrete Text-Defined Levels (2023, methodology): “In experiments, we fine-tune from the pre-trained weights of mPLUG-Owl2 (Ye et al., 2023b).”
MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models (2023, neutral): “Existing methods tend to use GPT or manual scoring [28, 33, 50], but there may be problems of inaccuracy and subjectivity.”
Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook (2024, neutral): “These models, among others [139,140], can be tailored and fine-tuned to specifically address the unique demands of tasks related to health care.”
A Survey on Vision--Language--Action Models for Embodied AI (2024, neutral): “For a more comprehensive comparison of human datasets, we refer interested readers to [217].”
Hallusionbench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models (2023, methodology): “Visual Illusion denotes the misinterpretation of accurate visual information.”
MMBench: Is Your Multi-modal Model an All-around Player? (2023, neutral): “…VLMs including model variants of OpenFlamingo [4], MiniGPT4 [56], InstructBLIP [12], LLaVA [32], IDEFICS [26], CogVLM [47], Qwen-VL [6], Yi-VL [2], mPLUG-Owl [50], InternLM-XComposer [13], and MiniCPM-V [39]; (c) Proprietary VLMs including Qwen-VL-[Plus/Max] [6], Gemini-Pro-V [44], and GPT-4v [37].”
MMMU: A Massive Multi-Discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI (2023, methodology): “Models such as LLaMA-Adapter [20, 81], mPlug-OWL [74, 75], SVIT [82], LRV-Instruction [39], and InstructBLIP [16] exemplify these developments.”
Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (2024, neutral): “In recent years, multimodal large language models (MLLMs) [60, 137, 246, 36, 35, 248, 140, 228, 192, 275, 143, 54, 170] have emerged as a pivotal technology in artificial intelligence, capable of processing and understanding information from multiple modalities such as text, images, and videos.”
Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action (2023, methodology): “We hypothesize that because it requires external knowledge, extensive language pre-training is important for this task, and therefore our reduced performance is since U NIFIED -IO 2 was not pre-trained as extensively on text as the dedicated language models used by Qwen-VL [12] and mPLUG-Owl2 [206].”
InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models (2025, neutral): “Multimodal large language models (MLLMs) [32, 66, 121, 21, 19, 123, 68, 114, 97, 136, 71, 31, 85, 117, 18, 89, 105, 69] have recently achieved or even surpassed human-level performance in a broad spectrum of tasks, underscoring their potential as a significant stride toward artificial general…”
Show-o: One Single Transformer to Unify Multimodal Understanding and Generation (2024, methodology): “…ii) When comparing with understanding only models including InstructBLIP (Dai et al., 2023), Qwen-VL-Chat (Bai et al., 2023), and mPLUG-Owl2 (Ye et al., 2024b) on multimodal understanding, our model with a much smaller model size also achieves competitive performance on POPE, MME, Flickr30k…”
MM-LLMs: Recent Advances in MultiModal Large Language Models (2024, neutral): “5 (Liu et al., 2023d), MiniGPT-v2 (Chen et al., 2023d), Fuyu-8B (Bavishi et al., 2023), CogVLM(Wang et al., 2023b), mPLUG-Owl2 (Ye et al., 2023c), Monkey (Li et al., 2023l), Volcano (Lee et al., 2023), DRESS (Chen et al., 2023i), LION (Chen et al., 2023c), DocPedia( I D ) (Feng et al., 2023),…”