Intern VL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks

By Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Zhong Muyan, Qinglong Zhang, Xizhou Zhu, Lewei Lu and 5 more. Computer Vision and Pattern Recognition, 2023

3,154 citations recorded by the source corpus.

The exponential growth of large language models (LLMs) has opened up numerous possibilities for multi-modal AGI systems. However, the progress in vision and vision-language foundation models, which are also critical elements of multi-modal AGI, has not kept pace with LLMs. In this work, we design a large-scale vision-language foun-dation model (Intern VL), which scales up the vision foun-dation model to 6 billion parameters and progressively aligns it with the LLM, using web-scale image-text data from various sources. This model can be broadly applied to and achieve state-of-the-art performance on 32 generic visual-linguistic benchmarks including visual perception tasks such as image-level or pixel-level recognition, vision-language tasks such as zero-shot image/video classification, zero-shot image/video-text retrieval, and link with LLMs to create multi-modal dialogue systems. It has powerful visual capabilities and can be a good alternative to the ViT-22B. We hope that our research could contribute to the development of multi-modal large models.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/6a33e58ef961a3a0a5657518b2be86395eb7c8d0

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Hallucination of Multimodal Large Language Models: A Survey (2024, neutral): “Unlearning refers to a technique designed to induce a model to ’forget’ specific behaviors or data, primarily through the application of gradient ascent methods [9].”
Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (2024, neutral): “In terms of data scale, the number of samples grows from 5 TextCaps (en) [211], ShareGPT4o (en & zh) [35], InternVL-SA-1B-Caption (en & zh) [36], Captioning NewYorkerCaptionContest (en) [88], MMInstruct (en & zh) [155] VQAv2 (en) [74], GQA (en) [98], OKVQA (en) [178], Visual7W (en) [317],…”
ShareGPT4Video: Improving Video Understanding and Generation with Better Captions (2024, background): “The vision-language community has recently made significant advancements in the field of large image-language models [36, 35, 37, 10, 13, 11, 58, 59, 2, 72, 33].”
LLaVA-OneVision: Easy Visual Task Transfer (2024, positive): “In subsequent collections, we accumulated open-sourced datasets from the Internet and referred to the dataset collection processes of other advanced LMMs, such as Qwen-VL [8], DeepSeek-VL [89], Intern-VL [22], Vision-Flan [145], UReader [149], Idefics-2 (Cauldron) [63], and Cambrian.”
Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook (2024, neutral): “…has been made in the field of general-domain image-text M-LLMs through the development of models such as GLaMM [71], Qwen-VL [72], SpatialVLM [73], InternVL [74], Osprey [75], Vary [76], ShareGPT4V [77], OtterHD [78], LION [79], SPHINX [80], BLIVA [81], SVIT [82], LLaVA [46], and CoVLM [83].”
Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning (2024, background): “In contrast, most MLLMs process aligned image contexts in a fixed-grain manner with a large amount of computation ( e.g., CLIP [50], EVA2-CLIP [54], InternVL [9]).”
InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency (2025, positive): “5, an advanced family of InternVL series [13, 14, 15, 37, 79, 80, 187] with stronger capabilities in versatility, reasoning, and efficiency.”
MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention (2024, neutral): “Recent studies [WWL + 24] have analyzed sparse attention patterns in multi-modal LLMs, revealing the presence of vertical and slash patterns in models like LLaVA [LLWL24] and In-ternVL [CWW + 24].”
LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding (2024, background): “…Multimodal Large Language Models (MLLMs) (Alayrac et al., 2022; Li et al., 2023b; Zhu et al., 2023; Liu et al., 2024c; Ye et al., 2023; Bai et al., 2023; Chen et al., 2023c; Dong et al., 2024) have demonstrated exceptional competencies in tasks such as captioning and visual question-answering.”
Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces (2024, neutral): “Multimodal Large Language Models (MLLMs) [1, 4, 15, 34, 42, 49, 49, 78], which integrate language and vision, exhibit strong capacities to think and reason in open-ended dialog and practical tasks like web agents [21, 28, 33, 35].”
SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuning (2024, neutral): “5-VL [2], GLM4-V [19], InternVL3 [65], and DiT models [45] , can all satisfy to this trend.”
Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction (2024, neutral): “Zero-shot generalization [55] refers to the ability of a model, particularly a Large Language Model, to perform tasks that it has not been explicitly trained on.”
Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation (2024, neutral): “…understanding component, a stronger vision encoder can be chosen without worrying about whether the encoder is capable of handling vision generation tasks, such as EVA-CLIP [71], InternViT [13], etc. (2) To handle high-resolution images, dynamic high-resolution techniques [48] can be used.”
A Survey on Hallucination in Large Vision-Language Models (2024, methodology): “However, the restricted quantity of these tokens prevents encoding all the information present in images during the alignment process [ Yin et al. , 2023a; Chen et al. , 2023c ] , and consequent information loss raises the risk of hallucinations.”
InternVideo2: Scaling Foundation Models for Multimodal Video Understanding (2024, methodology): “Specifically, we adopt InternVL-6B [Chen et al., 2023c] and VideoMAEv2-g [Wang et al., 2023b] to transfer unmasked knowledge via simple projection layers.”
SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference (2024, neutral): “To combine visual signals with textual semantics, the mainstream practice in VLMs (Team et al., 2023; Bai et al., 2023b; Chen et al., 2023; Li et al., 2024b; 2023a) employs sequential visual representation, where images are extracted into vision tokens and sent into an LLM decoder.”
VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding (2025, positive): “For Object365, COCO-2017, and SA-1B datasets, we combined the original image annotations with InternVL2-26B [31] to recaption and generate detailed image captions.”