How far are we to GPT-4V? Closing the gap to commercial multimodal models with open-source suites

By Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma and 18 more. Science China Information Sciences, 2024

1,231 citations recorded by the source corpus.

In this paper, we introduce InternVL 1.5, an open-source multimodal large language model (MLLM) to bridge the capability gap between open-source and proprietary commercial models in multimodal understanding. We introduce three simple improvements. (1) Strong vision encoder: we explored a continuous learning strategy for the large-scale vision foundation model — InternViT-6B, boosting its visual understanding capabilities, and making it can be transferred and reused in different LLMs. (2) Dynamic high-resolution: we divide images into tiles ranging from 1 to 40 of 448×448 pixels according to the aspect ratio and resolution of the input images, which supports up to 4K resolution input. (3) High-quality bilingual dataset: we carefully collected a high-quality bilingual dataset that covers common scenes, document images, and annotated them with English and Chinese question-answer pairs, significantly enhancing performance in optical character recognition (OCR) and Chinese-related tasks. We evaluate InternVL 1.5 through a series of benchmarks and comparative studies. Compared to both open-source and proprietary commercial models, InternVL 1.5 shows competitive performance, achieving state-of-the-art results in 8 of 18 multimodal benchmarks. Code and models are available at https://github.com/OpenGVLab/InternVL.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/887306ee7ebd9eebd7faea24106ec8e8f1a50987

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

R1-VL: Learning to Reason with Multimodal Large Language Models via Step-Wise Group Relative Policy Optimization (2025, neutral): “Multimodal large language models (MLLMs) have achieved significant progress in vision-language understanding [1, 7, 13, 16, 19, 34, 40, 46].”
Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation (2024, neutral): “Tokens can be compressed to save computational cost, for instance, using pixel shuffle operation [12].”
Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone (2024, positive): “5-Vision with the following baseline methods: Llava Interleave-Qwen 7B [LZZ + 24], InternVL2 4B and 8B [CWT + 24], Gemini 1.5 Flash [TAB + 23], GPT-4o-mini, Claude 3.5 Sonnet [Ant24], Gemini 1.5 Pro [TAB + 23], and GPT-4O.”
LongVILA: Scaling Long-Context Visual Language Models for Long Videos (2024, neutral): “LongVILA-8B, utilizing 256 frames, achieves an overall score of 50.5.”
MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark (2024, background): “…is motivated by several key observations, including the text-only solvability of some existing benchmark questions, the limited option space in multiple-choice formats (Wang et al., 2024), and the need to challenge models’ ability to jointly understand different modalities in a more integrated way.”
Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis (2024, neutral): “5 [11].”
MLVU: A Comprehensive Benchmark for Multi-Task Long Video Understanding (2024, neutral): “Recent studies have made significant progresses in this field by integrating LLM backbones with visual encoders and adapters, and fine-tuning the entire architecture through visual instruction [25; 55; 7].”
Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs (2025, neutral): “…input modality, numerous vision-language models have been developed, including the LLava series [LLWL24, LLL + 24, LZG + 24], QWenVL series [BBY + 23, WBT + 24], InternVL series [CWT + 24, CWW + 24, CWC + 24], InternLM-XComposer series [ZDW + 23, DZZ + 24b], Molmo [DCL + 24], and NVLM [DLW + 24].”
LVBench: An Extreme Long Video Understanding Benchmark (2024, neutral): “Models like InternVL2 [6], LLaVA-Next [50], and CogVLM2 [12] unify high-resolution images and videos as multi-frame sequences, enabling seamless processing of both modalities.”
Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs (2024, methodology): “Our current Cambrian-1 model uses a moderate number of visual tokens and does not adopt the any-resolution strategy [25, 70, 74] to handle ultra high-resolution images or those with extreme aspect ratios, which require a larger number of visual tokens.”
InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency (2025, neutral): “5, an advanced family of InternVL series [13, 14, 15, 37, 79, 80, 187] with stronger capabilities in versatility, reasoning, and efficiency.”
VLMEvalKit: An Open-Source ToolKit for Evaluating Large Multi-Modality Models (2024, background): “VLMEvalKit supports most mainstream commercial APIs including GPT-4o, Gemini, GLM-4v, etc. , as well as over 50 open-source LMMs including LLaVA [29], InternVL [9], Idefics [22], etc. or LMMs.”
VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model (2025, positive): “To-day, some of the most widely adopted open-source VLM series include LLaVA[25, 30, 31], QwenVL[6, 50], and In-ternVL [10, 11].”
InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models (2025, neutral): “Yet, the majority of leading MLLMs—both open-source and proprietary—are adapted from text-only large language models through sophisticated multi-stage pipelines [21, 19, 18, 5, 121, 7].”
VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding (2025, neutral): “For scene images, we include COCO-2017 [66], Object365 [56], SA-1B [57], ShareGPT4o [53], ShareGPT4V [63], DenseFusion [64], and LLaVA-Recap (LCS-558K) [28].”
OS-ATLAS: A Foundation Action Model for Generalist GUI Agents (2024, methodology): “InternVL-2-4B employs AnyRes (Liu et al., 2024; You et al., 2024) to resize images and segment larger images into smaller patches, which are then encoded independently using vision encoders.”
Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (2024, neutral): “…(en) [233], MP-Docmatix (en) [121] Type: Video Datasets Vript (en & zh) [269], OpenVid (en) [190], Mementos (en) [254], ShareGPT4o-Video (en & zh) [35], Captioning ShareGPT4Video (en & zh) [30], VideoGPT+ (en) [174] VideoChat2-IT (en & zh) [130, 131], EgoTaskQA (en) [99], NTU RGB+D (en) [152],…”