LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Models

By Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Yuhao Dong, Shuai Liu, Yuanhan Zhang, Jingkang Yang and 2 more. North American Chapter of the Association for Computational Linguistics, 2024

331 citations recorded by the source corpus.

The advances of large foundation models necessitate wide-coverage, low-cost, and zero-contamination benchmarks. Despite continuous exploration of language model evaluations, comprehensive studies on the evaluation of Large Multi-modal Models (LMMs) remain limited. In this work, we introduce LMMS-EVAL, a unified and standardized multimodal benchmark framework with over 50 tasks and more than 10 models to promote transparent and reproducible evaluations. Although LMMS-EVAL offers comprehensive coverage, we find it still falls short in achieving low cost and zero contamination. To approach this evaluation trilemma, we further introduce LMMS-EVAL LITE, a pruned evaluation toolkit that emphasizes both coverage and efficiency. Additionally, we present Multimodal LIVEBENCH that utilizes continuously updating news and online forums to assess models'generalization abilities in the wild, featuring a low-cost and zero-contamination evaluation approach. In summary, our work highlights the importance of considering the evaluation trilemma and provides practical solutions to navigate the trade-offs in evaluating large multi-modal models, paving the way for more effective and reliable benchmarking of LMMs. We opensource our codebase and maintain leaderboard of LIVEBENCH at https://github.com/EvolvingLMMs-Lab/lmms-eval and https://huggingface.co/spaces/lmms-lab/LiveBench.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/84da5687bdb173c1ca00078c17317f1532e7a033

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Bo Li
2. Ziwei Liu
3. Renrui Zhang
4. Yuanhan Zhang
5. Chunyuan Li
6. Chunyuan Li
7. Kaichen Zhang
8. Jianfeng Gao

Citations in the corpus

• VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding (2025, neutral): “scores, and we also align our evaluation protocols with other evaluation toolkits, such as lmms-eval [123, 124] and VLMEvalKit [125].”
• LLaVA-OneVision: Easy Visual Task Transfer (2024, positive): “To validate the capabilities in real-world scenarios, we utilize several widely-adopted benchmarks, including RealworldQA [140], Vibe-Eval [111], MM-LiveBench [160], and LLaVA-Bench-Wilder [65].”
• OCRBench: on the hidden mystery of OCR in large multimodal models (2023, positive): “[31, 32] have developed effective systems to assess LMMs. [15, 33, 22] introduces GPT-4 or human evaluation to assess the output of LMMs. MMBench [34] evaluates LMM using multiple-choice questions across various dimensions of ability.”