VLMEvalKit: An Open-Source ToolKit for Evaluating Large Multi-Modality Models

By Haodong Duan, Junming Yang, Yu Qiao, Xinyu Fang, Lin Chen, Mo Li, Yuan Liu, Xiao-wen Dong, Yuhang Zang, Pan Zhang and 3 more. ACM Multimedia, 2024

565 citations recorded by the source corpus.

We present VLMEvalKit: an open-source toolkit for evaluating large multi-modality models based on PyTorch. The toolkit aims to provide a user-friendly and comprehensive framework for researchers and developers to evaluate existing multi-modality models and publish reproducible evaluation results. In VLMEvalKit, we implement over 70 different large multi-modality models, including both proprietary APIs and open-source models, as well as more than 20 different multi-modal benchmarks. By implementing a single interface, new models can be easily added to the toolkit, while the toolkit automatically handles the remaining workloads, including data preparation, distributed inference, prediction post-processing, and metric calculation. Although the toolkit is currently mainly used for evaluating large vision-language models, its design is compatible with future updates that incorporate additional modalities, such as audio and video. Based on the evaluation results obtained with the toolkit, we host a comprehensive leaderboard (https://huggingface.co/spaces/opencompass/open_vlm_leaderboard) to track the progress of multi-modality learning research. The toolkit is released on GitHub (https://github.com/open-compass/VLMEvalKit) and is actively maintained.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/24a91683b4011fd24d9593ae9a48783234bc9819

Citations in the corpus

• InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency (2025, unknown): “…Hal-lusionBench [41], MVBench [63], VideoMME [35], MLVU [186], (2) Reasoning Tasks : MMMU [170], MathVista [76], MathVision [76], MathVerse [175], DynaMath [189], WeMath [100], OlympiadBench [43], LogicVista [153]; (3) Text-Centric Tasks : MATH500 [65], AIME24 [84], AIME25 [85], GPQA [106], [31].”
• Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (2024, unknown): “To comprehensively evaluate InternVL’s performance on multimodal tasks, we employ a diverse set of benchmarks, including both well-established classic datasets and newly introduced ones provided by VLMEvalKit [63].”
• LlaVA-CoT: Let Vision Language Models Reason Step-By-Step (2024, unknown): “To ensure fairness and reproducibility, all evaluations are conducted using VLMEvalKit [15], an open-source evaluation toolkit for large vision-language models.”
• ShareGPT4Video: Improving Video Understanding and Generation with Better Captions (2024, unknown): “In Section 2.1, we describe our methods for collecting and filtering video data.”
• InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models (2025, unknown): “…understanding, multi-image understanding, real-world comprehension, comprehensive multimodal evaluation, multimodal hallucination evaluation, visual grounding, multimodal multilingual understanding, video understanding, and other multimodal tasks, most of which were tested using VLMEvalKit [33].”
• Are We on the Right Way for Evaluating Large Vision-Language Models? (2024, unknown): “In Section 4.3, we introduce two benchmark-specific metrics developed to evaluate the degree of data leakage as well as the actual performance gains in multimodal capabilities from the multi-modal training.”
• VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding (2025, unknown): “scores, and we also align our evaluation protocols with other evaluation toolkits, such as lmms-eval [123, 124] and VLMEvalKit [125].”