InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Composition
By Pan Zhang, Xiaoyi Wang, Yuhang Cao, Chao Xu, Linke Ouyang, Zhiyuan Zhao, Shuangrui Ding, Songyang Zhang, Haodong Duan, Hang Yan and 8 more. arXiv.org, 2023
354 citations recorded by the source corpus.
We propose InternLM-XComposer, a vision-language large model that enables advanced image-text comprehension and composition. The innovative nature of our model is highlighted by three appealing properties: 1) Interleaved Text-Image Composition: InternLM-XComposer can effortlessly generate coherent and contextual articles that seamlessly integrate images, providing a more engaging and immersive reading experience. Simply provide a writing instruction, and our system will generate the corresponding manuscript. It can intelligently identify the areas in the text where images would enhance the content and automatically insert the most appropriate visual candidates. 2) Comprehension with Rich Multilingual Knowledge: The text-image comprehension is empowered by training on an extensive multi-modal multilingual database with carefully crafted strategies, resulting in a deep understanding of visual content. 3) State-of-the-art Performance: Our model consistently achieves state-of-the-art results across various mainstream benchmarks for vision-language foundational models, including MME Benchmark, MMBench, MMBench-CN, Seed-Bench, CCBench (Chinese Cultural Benchmark), QBench and Tiny LVLM. Owing to the absence of established metrics for quantitatively assessing text-image composition, we have devised a robust evaluation procedure that comprises both human and GPT4-Vision (GPT4-V) to ensure reliability. Notably, our InternLM-XComposer achieves competitive text-image composition scores compared to public solutions, including GPT4-V and GPT3.5. Collectively, InternLM-XComposer seamlessly blends advanced text-image comprehension and composition, revolutionizing vision-language interaction and offering new insights and opportunities. The InternLM-XComposer model series are publicly available at https://github.com/InternLM/InternLM-XComposer.
Subfields: Computer Vision and Pattern Recognition
View Paper: https://www.semanticscholar.org/paper/c1e450284e7d6cac1855330a1197df8537df653f
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
Citations in the corpus
- mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models (2024, methodology): “InternLM-XComposer (Zhang et al., 2023) and IDEFICS2 (Laurençon et al., 2024) also use the similar method.”
- SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generation (2024, neutral): “In recent years, Multimodal Large Language Models (MLLMs) [1, 2, 3, 4, 5, 6, 7, 8] have demonstrated exceptional capabilities in comprehending multimodal data through leveraging the strong generality of LLMs [9, 10, 11].”
- Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs (2025, neutral): “…input modality, numerous vision-language models have been developed, including the LLava series [LLWL24, LLL + 24, LZG + 24], QWenVL series [BBY + 23, WBT + 24], InternVL series [CWT + 24, CWW + 24, CWC + 24], InternLM-XComposer series [ZDW + 23, DZZ + 24b], Molmo [DCL + 24], and NVLM [DLW + 24].”
- Generative Multimodal Models are In-Context Learners (2023, background): “A school of successive efforts [69, 76, 84, 90, 91] further improves visual instruction tuning with better overall training pipelines [6, 40], grounding annotations [14, 15, 85, 92], and extra tasks [6].”
- ShareGPT4V: Improving Large Multi-Modal Models with Better Captions (2023, methodology): “Following the evolution, modality unification via LLMs becomes the inevitable tendency, and visual-aligned multi-modal LLMs[3, 5, 10, 30, 31, 35, 57, 60–62] have witnessed ever-changing advances in recent days.”
- OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation (2023, neutral): “Recent advancements in multi-modal large language models (MLLMs) [1, 5, 9, 10, 30, 31, 44, 48] has greatly ele-* Please describe this image in detail.”
- SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents (2024, background): “Recently, researchers have invested tremendous effort in constructing LVLMs capable of jointly processing image and text (Liu et al., 2023a; Zhu et al., 2023; Ye et al., 2023b; Li et al., 2023; Zhang et al., 2023a).”
- Are We on the Right Way for Evaluating Large Vision-Language Models? (2024, methodology): “Encouraged by the rapid development of large language models (LLMs) [47, 4, 8, 9, 13, 1, 43], integrating visual modality into LLMs to enhance models’ interactivity capabilities has witnessed ever-changing advances in recent days [54, 26, 24, 11, 52, 2, 48, 31, 5, 12].”
- InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model (2024, methodology): “…improve the MLLM in different dimensions, such as instruction tuning [11, 49, 98], efficient fine-tuning [33], high-resolution image inputs [6, 82, 83], hallucination mitigation [34, 87, 99], image generation [23, 30, 74, 89], 3D understanding [63] and image-text comprehension and composition [95].”
- Intern VL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks (2023, methodology): “Recent advancements have seen the creation of vision large language models (VLLMs) [3, 21, 56, 59, 62, 66, 100, 121, 128, 130, 136, 138, 141, 142, 148], which aim to enhance language models with the capability to process and interpret visual information.”
- DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models (2024, background): “Recently, there has been a surge in research on large Vision-Language Models (VLMs), exemplified by works such as MiniGPT-4 [57], LLaVA [32], Qwen-VL [1], and others [12, 44, 54, 55].”