LVLM-EHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models

By Peng Xu, Wenqi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu, Meng Lei, Fanqing Meng, Siyuan Huang, Yu Qiao, Ping Luo. 2023

280 citations recorded by the source corpus.

Large Vision-Language Models (LVLMs) have recently played a dominant role in multimodal vision-language learning. Despite the great success, it lacks a holistic evaluation of their efficacy. This paper presents a comprehensive evaluation of publicly available large multimodal models by building an LVLM evaluation Hub (LVLM-eHub). Our LVLM-eHub consists of 13 representative LVLMs such as InstructBLIP and LLaVA, which are thoroughly evaluated by a quantitative capability evaluation and an online arena platform. The former evaluates five categories of multimodal capabilities of LVLMs such as visual question answering and object hallucination on 42 in-domain text-related visual benchmarks, while the latter provides the user-level evaluation of LVLMs in an open-world question-answering scenario. The study investigates how specific features of LVLMs such as model configurations, modality alignment mechanisms, and training data affect the multimodal understanding. By conducting a comprehensive comparison of these features on quantitative and arena evaluation, our study uncovers several innovative findings, which establish a fundamental framework for the development and evaluation of innovative strategies aimed at enhancing multimodal techniques.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Hongsheng Li
2. Yu Qiao
3. Peng Gao
4. Ping Luo
5. Renrui Zhang
6. Graham Neubig
7. Dahua Lin
8. Pan Lu

In Computer Vision and Pattern Recognition

1. Hongsheng Li
2. Yu Qiao
3. Xiaogang Wang
4. Luc Van Gool
5. Chen Change Loy
6. Dacheng Tao
7. Yi Yang
8. Ming-Hsuan Yang

In Multimodal Learning

1. Peng Gao
2. Hongsheng Li
3. Yu Qiao
4. Renrui Zhang
5. Pan Lu
6. Huan Sun
7. Ying Shan
8. Bo Li

In Evaluation & Benchmarks

1. Dongfu Jiang
2. Huan Sun
3. Ying Shan
4. Bo Li
5. Yu Qiao
6. Peng Gao
7. Rui Zhao
8. Wenqi Shao

Citations in the corpus

• LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention (2023, background): “In addition to instruction-following models, our zero-initialized attention can be generalized to other vision and language models for parameter-efficient fine-tuning.”
• MMBench: Is Your Multi-modal Model an All-around Player? (2023, background): “The first approach involves utilizing existing public datasets [19, 9] for objective evaluation, while the second approach employs human annotators [49, 48] to perform subjective evaluations.”
• MVBench: A Comprehensive Multi-modal Video Understanding Benchmark (2023, unknown): “In response to this need, a number of benchmarks have been launched [17, 45, 49, 90, 97], by evaluating MLLMs with Question Answering (QA) formulation of various perception tasks.”
• SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension (2023, methodology): “With the rapid development of Multimodal Large Language Models (MLLMs), some concurrent works [23, 24, 25, 26] propose various benchmarks for evaluating MLLMs.”
• A Survey on Evaluation of Large Language Models (2023, unknown): “LVLM-eHub [218] undertakes an exhaustive evaluation of LVLMs using an online competitive platform and quantitative capacity assessments.”
• Visual Instruction Tuning with Polite Flamingo (2023, methodology): “As a result, the InstructBLIP model, the current SoTA model on a wide range of vision-language benchmarks, ranked second to last (Li et al. 2023a) in Multi-Modality Arena (Xu et al. 2023), a user rating-based evaluation platform of multi-modal LLMs.”
• OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models (2023, background): “Furthermore, the language understanding capabilities inherent in LLMs can be successfully transferred into multimodal models (Mu et al., 2023; Xu et al., 2023; Zhang et al., 2023a; Huang et al., 2024; 2023).”
• GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interest (2023, unknown): “These models achieve amazing image-level multimodal abilities, while several benchmarks such as LVLM-eHub (Xu et al., 2023) and MMBench (Liu et al., 2023c) find that these models still have performance bottlenecks when need to be under specific region reference.”
• MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts (2023, background): “In recent times, there have been several works, such as Visit-Bench, LVLM-eHub, MMBench (Bitton et al., 2023; Yu et al., 2023; Xu et al., 2023; Shao et al., 2023), that assess their instruction-following and reasoning capabilities.”
• MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities (2023, unknown): “Our MM-Vet is most related to the concurrent evaluation studies [23, 49, 41, 73] such as MME and MMBench, which design comprehensive evaluation samples to facilitate the LMM evaluation.”
• Monkey: Image Resolution and Text Label are Important Things for Large Multi-Modal Models (2023, unknown): “Despite these advancements, challenges remain in extracting finer image features, as noted by [30, 51], which indicate the need for ongoing development in the field.”
• VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use (2023, background): “Table 1 presents a summary of our contributions in comparison to the recent works [32, 14, 17, 33, 34, 35] in the evaluation of multimodal chatbots.”
• A survey on multimodal large language models (2023, unknown): “To this end, some efforts have endeavored to develop new benchmarks specially designed for MLLMs [39,40,72].”
• Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic (2023, methodology): “8) results on LVLM-eHub toolbox (Xu et al., 2023) for easy comparison.”
• SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities (2024, unknown): “Many prior works have focused on benchmarking VLMs [58, 66], considering tasks like VQA (e.g. VQAv2 [28], OK-VQA [47], COCO [41], or Visual Genome [37]).”
• MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems? (2024, background): “Despite comprehensive benchmarks [17, 40, 33, 59] on general visual instruction-following scenarios, the specific potential of MLLMs for visual mathematical problem-solving remains under-explored.”
• MMMU: A Massive Multi-Discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI (2023, methodology): “More holistic evaluations have been conducted as well, such as LAMM [76], LVLM-eHub [72], SEED [30], MMBench [43], and MM-Vet [79].”
• MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark (2024, methodology): “To address this, recent benchmarks such as LAMM (Yin et al., 2023b), LVLM-eHub (Xu et al., 2023), SEED (Li et al., 2024a), MMBench (Liu et al., 2023d),CV-Bench (Tong et al., 2024a), MM-Vet (Yu et al., 2024), Mantis (Jiang et al., 2024), and BLINK (Fu et al., 2024) have emerged, covering aspects from”