SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension
By Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, Ying Shan. 2023
964 citations recorded by the source corpus.
Based on powerful Large Language Models (LLMs), recent generative Multimodal Large Language Models (MLLMs) have gained prominence as a pivotal research area, exhibiting remarkable capability for both comprehension and generation. In this work, we address the evaluation of generative comprehension in MLLMs as a preliminary step towards a comprehensive assessment of generative models, by introducing a benchmark named SEED-Bench. SEED-Bench consists of 19K multiple choice questions with accurate human annotations (x 6 larger than existing benchmarks), which spans 12 evaluation dimensions including the comprehension of both the image and video modality. We develop an advanced pipeline for generating multiple-choice questions that target specific evaluation dimensions, integrating both automatic filtering and manual verification processes. Multiple-choice questions with groundtruth options derived from human annotation enables an objective and efficient assessment of model performance, eliminating the need for human or GPT intervention during evaluation. We further evaluate the performance of 18 models across all 12 dimensions, covering both the spatial and temporal understanding. By revealing the limitations of existing MLLMs through evaluation results, we aim for SEED-Bench to provide insights for motivating future research. We will launch and consistently maintain a leaderboard to provide a platform for the community to assess and investigate model capability.
Subfields: Artificial Intelligence, General Social Sciences, Large Language Models, Multimodal Learning, Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Furu Wei
- Yu Qiao
- Percy Liang
- Luke S. Zettlemoyer
- Hongsheng Li
- Caiming Xiong
In General Social Sciences
In Large Language Models
In Multimodal Learning
In Evaluation & Benchmarks
Citations in the corpus
- Improved Baselines with Visual Instruction Tuning (2023, methodology): “SEED-Bench [26] evaluates model’s performance on both images and videos with multiple choice, and we sample the frame in the middle to evaluate the accuracy on videos.”
- Hallusionbench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models (2023, methodology): “To address these challenges, research communities have introduced a series of benchmarks, including MME [15], MMBench [34], MM-Vet [54], SEED-Bench [21], GAVIE [29], and LAMM-Bench [14].”
- Generative Multimodal Models are In-Context Learners (2023, methodology): “The evaluation also encompassed recent benchmarks for large multimodal models, including SEED-Bench [39], MM-Vet [88], TouchStone [7] and MMMU [89].”
- Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action (2023, background): “Vision-language results on nine tasks [1, 28, 59, 91, 124, 129, 130, 136, 209] and three evaluation-only benchmarks [106, 113, 120].”
- BLINK: Multimodal Large Language Models Can See but Not Perceive (2024, background): “To appropriately position our paper, let us revisit how we currently evaluate perception through using multimodal LLMs [43,44,47,52,53,57,87].”
- Kosmos-2: Grounding Multimodal Large Language Models to the World (2023, background): “On SEED-Bench, we compare K OSMOS -2 with popular MLLMs including MiniGPT4 (Zhu et al., 2023), LLaVA (Liu et al., 2023a), BLIP-2 (Li et al., 2023b), InstructBLIP (Dai et al., 2023), MultiModal-GPT (Gong et al., 2023), mPLUG-Owl (Ye et al., 2023), VideoChat (Li et al., 2023c) and Video-ChatGPT (Maaz ”
- SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generation (2024, positive): “We evaluate SEED-X-I on benchmarks specifically designed for evaluating MLLMs, since recent work [31, 21] point out that traditional VQA benchmarks are not tailored for evaluating MLLMs with open-from output.”
- OtterHD: A High-Resolution Multi-modality Model (2023, methodology): “Benchmarks like MME [21], MMBench [34], and SEED-Bench [26] do evaluate the perceptual abilities of LMMs, yet they do not adequately focus on the nuanced perception of smaller objects.”
- InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Composition (2023, background): “It consistently achieves state-of-the-art performance across various leading benchmarks for visionlanguage large models, encompassing MME Benchmark [18, 71], MMBench [72], Seed-Bench [29] in English, and MMBench-CN [72], CCBench (Chinese Cultural Benchmark) [72] for evaluations in Chinese.”
- Emu3: Next-Token Prediction is All You Need (2024, methodology): “6 [56], on a series of public vision-language benchmarks, including SEED-Bench [45], RealWorldQA [91], OCRBench [59], etc .”
- What matters when building vision-language models? (2024, positive): “Similarly to (Sanh et al., 2022; Wei et al., 2022; Bach et al., 2022; Dai et al., 2023; Li et al., 2023), each dataset is prompted into a shared question/answer format.”
- DeepSeek-VL: Towards Real-World Vision-Language Understanding (2024, methodology): “…datasets utilized in our study encompass a diverse range of multi-modality and language data sources, including well-known open-source shared gpt4v datasets such as ShareGPT4V (Chen et al., 2023), LAION-GPTV (LAION, 2023), LVIS-Instruct4V (Wang et al., 2023a), textOCR-GPT4V (Carter, 2024), LLaVA1.”
- Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond (2023, methodology): “In SEED-Bench, we have found that our model’s visual capabilities can be effectively transferred to video tasks by simply sampling four frames.”
- Are We on the Right Way for Evaluating Large Vision-Language Models? (2024, methodology): “To address this issue, comprehensive multi-modal benchmarks have been constructed [26, 21, 51, 14, 27, 7, 46].”
- CogVLM: Visual Expert for Pretrained Language Models (2023, neutral): “We conduct multidimensional evaluations of the models on datasets including MM-Vet [Yu et al., 2023], MMBench [Liu et al., 2023g], SEED-Bench [Li et al., 2023a], LLaVA-Bench [Liu et al., 2023c], POPE [Li et al., 2023c], MMMU [Yue et al., 2023] and MathVista [Lu et al., 2023].”
- InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model (2024, neutral): “We report results in Math-Vista [52], MMMU [91], AI2D [40], MME [27], MM-Bench [51], MMBench-Chinese [51], SEED-Bench (Image) [41], LLaVA-Bench (In-the-Wild) [49], QBench [85], MM-Vet [90], HallusionBench [31], ChartQA [56], and POPE [45].”
- MVBench: A Comprehensive Multi-modal Video Understanding Benchmark (2023, methodology): “Recently, several attempts have tried to evaluate MLLMs on temporal perception in videos [37, 51, 61, 87].”