InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model

By Xiao-wen Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao and 13 more. arXiv.org, 2024

387 citations recorded by the source corpus.

We introduce InternLM-XComposer2, a cutting-edge vision-language model excelling in free-form text-image composition and comprehension. This model goes beyond conventional vision-language understanding, adeptly crafting interleaved text-image content from diverse inputs like outlines, detailed textual specifications, and reference images, enabling highly customizable content creation. InternLM-XComposer2 proposes a Partial LoRA (PLoRA) approach that applies additional LoRA parameters exclusively to image tokens to preserve the integrity of pre-trained language knowledge, striking a balance between precise vision understanding and text composition with literary talent. Experimental results demonstrate the superiority of InternLM-XComposer2 based on InternLM2-7B in producing high-quality long-text multi-modal content and its exceptional vision-language understanding performance across various benchmarks, where it not only significantly outperforms existing multimodal models but also matches or even surpasses GPT-4V and Gemini Pro in certain assessments. This highlights its remarkable proficiency in the realm of multimodal understanding. The InternLM-XComposer2 model series with 7B parameters are publicly available at https://github.com/InternLM/InternLM-XComposer.

Subfields: Computer Vision and Pattern Recognition, Literature and Literary Theory

View Paper: https://www.semanticscholar.org/paper/bce43cb9af37a0c8d90f8cadaebd6bb002685edd

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Xiaogang Wang
2. Luc Van Gool
3. Peng Gao
4. Hongsheng Li
5. Dahua Lin
6. Dacheng Tao
7. Yu Qiao
8. Chen Change Loy

In Literature and Literary Theory

1. Noah A. Smith

Citations in the corpus

• MMBench: Is Your Multi-modal Model an All-around Player? (2023, methodology): “During the instruction tuning, Low-Rank Adaptation (LoRA [22]) has been adopted by recent works [49, 13, 11] on language models to achieve better performance on multimodal understanding.”
• How far are we to GPT-4V? Closing the gap to commercial multimodal models with open-source suites (2024, neutral): “Over the past year, many open-source MLLMs have become well-known, including the LLaVA series [58– 60], MiniGPT-4 [137], VisionLLM [111], Qwen-VL [7], CogVLM [112], Shikra [17], and others [20, 25, 86, 114].”
• SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models (2023, background): “LLaMA-2-70B shows a marked improvement in the few-shot setting over the zero-shot setting, increasing from 2.41% to 8.40%.”
• ShareGPT4Video: Improving Video Understanding and Generation with Better Captions (2024, background): “Recent advancements in multi-modal learning, driven by large language models, have led to progress in image-text dialogue[35, 10, 16, 56, 63, 13, 2, 65] and text-to-image generation tasks[3, 52, 6, 53, 70, 8].”
• DeepSeek-VL: Towards Real-World Vision-Language Understanding (2024, methodology): “As illustrated in Figure 7, DeepSeek-VL is judged to be superior in over 60% of instances when compared to open-source multimodal models, including Fuyu-8B, CogVLM-17B, and InternLM-XComposer2-VL.”
• Are We on the Right Way for Evaluating Large Vision-Language Models? (2024, background): “[12] achieves an impressive score of 55.4%.”
• MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems? (2024, methodology): “5 [37] (Vicuna-13B [13]), LLaVA-NeXT [38] (Vicuna-13B), SPHINX-MoE [21] (Mixtral-8×7B [28]), SPHINX-Plus (LLaMA2-13B [56]), InternLM-XComposer2 [16] (InternLM2-7B [54]), LLaMA-Adapter V2 [20] (LLaMA-7B [55]), ImageBind-LLM [24] (LLaMA-7B), MiniGPT-v2 [11] (LLaMA2-7B), mPLUG-Owl2 [62] (LLaMA-7B),…”
• Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs (2025, neutral): “…input modality, numerous vision-language models have been developed, including the LLava series [LLWL24, LLL + 24, LZG + 24], QWenVL series [BBY + 23, WBT + 24], InternVL series [CWT + 24, CWW + 24, CWC + 24], InternLM-XComposer series [ZDW + 23, DZZ + 24b], Molmo [DCL + 24], and NVLM [DLW + 24].”
• VLMEvalKit: An Open-Source ToolKit for Evaluating Large Multi-Modality Models (2024, methodology): “InternLM-XComposer2 [13] impresses with its capabilities in multi-modal math and diagram understanding, while two LMMs based on GLM [15] notably excel on OCRBench and MMVet.”
• VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs (2024, background): “Models capable of image recognition (Bai et al., 2023; Dong et al., 2024; Chen et al., 2023d; Liu et al., 2024c; Chen et al., 2023b; Lin et al., 2023c; Young et al., 2024) and photorealistic image generation (Esser et al., 2024; Saharia et al., 2022; Ramesh et al., 2021) have approached near-human…”
• OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation (2023, neutral): “Recent advancements in multi-modal large language models (MLLMs) [1, 5, 9, 10, 30, 31, 44, 48] has greatly ele-* Please describe this image in detail.”
• BLINK: Multimodal Large Language Models Can See but Not Perceive (2024, background): “…in the modern era of large language models (LLMs), we, as a community, have focused less on such perceptual tasks, and instead have developed new tasks, mostly expressed in natural language, emphasizing the vision-language connection learned by multimodal LLMs [3,6,18,20,24,26,50,51,56,62,71,77].”