BLINK: Multimodal Large Language Models Can See but Not Perceive

By Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A. Smith, Wei-Chiu Ma, Ranjay Krishna. European Conference on Computer Vision, 2024

543 citations recorded by the source corpus.

We introduce Blink, a new benchmark for multimodal language models (LLMs) that focuses on core visual perception abilities not found in other evaluations. Most of the Blink tasks can be solved by humans"within a blink"(e.g., relative depth estimation, visual correspondence, forensics detection, and multi-view reasoning). However, we find these perception-demanding tasks cast significant challenges for current multimodal LLMs because they resist mediation through natural language. Blink reformats 14 classic computer vision tasks into 3,807 multiple-choice questions, paired with single or multiple images and visual prompting. While humans get 95.70% accuracy on average, Blink is surprisingly challenging for existing multimodal LLMs: even the best-performing GPT-4V and Gemini achieve accuracies of 51.26% and 45.72%, only 13.17% and 7.63% higher than random guessing, indicating that such perception abilities have not"emerged"yet in recent multimodal LLMs. Our analysis also highlights that specialist CV models could solve these problems much better, suggesting potential pathways for future improvements. We believe Blink will stimulate the community to help multimodal LLMs catch up with human-level visual perception.

View Paper: https://www.semanticscholar.org/paper/09ff7cdc583b28ec63dfc85c7142f2cb0e5e4eef

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Bo Li
2. Mohit Bansal
3. Renrui Zhang
4. Yoshua Bengio
5. Graham Neubig
6. Percy Liang
7. Zhe Gan
8. Dacheng Tao

Citations in the corpus

• Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs (2024, background): “These concerns are not unfounded, as researchers have started to notice that visual grounding is becoming a bottleneck for applying MLLMs in some challenging real-world applications, despite significant progress in improving general capabilities [35, 117, 127].”
• LLaVA-OneVision: Easy Visual Task Transfer (2024, unknown): “They include the multi-image split of math QA benchmark MathVerse [164] and science QA benchmark SciVerse [34], multi-image perception benchmark BLINK [31], MMMU-(multi-image) [156] that contains all multi-image QA in MMMU, and MuirBench [135] spanning 12 diverse multi-image tasks.”
• Gemma 3 Technical Report (2025, unknown): “We will continue to adopt assessments and safety mitigations proportionate to the potential risks from our models, and will only share these with the community when we are confident that the benefits significantly outweigh the foreseeable risks.”
• Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (2024, unknown): “BLINK [70]: The BLINK benchmark evaluates the core visual perception capabilities of MLLMs through 14 tasks inspired by classic computer vision challenges.”
• InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency (2025, unknown): “1 (en) [71], MMStar [11], BLINK [36], HallusionBench [41], AI2D [55], OCRBench [72], MMVet [168], MME-RealWorld (en) [178], MVBench [63], VideoMME [35], MMMU [170], MathVista [76], MathVision [134], MathVerse [175], DynaMath [189], WeMath [100], Log-icVista [153], MATH500 [45], AIME24 [84], AIME25…”
• SpatialRGPT: Grounded Spatial Reasoning in Vision Language Model (2024, background): “This includes difficulties [8, 9] in distinguishing simple spatial concepts like "left" and "right," "above" and "below," as well as more complex relationships such as "behind" and "in front," "inside" and "outside," and "near" and "far."”
• Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Models (2024, methodology): “For computer vision, we tackle diverse tasks including (1) depth, (2) spatial reasoning, (3) jigsaw, (4) visual correspondence, (5) semantic correspondence, as well as questions from (6) the MMVP and (7) the V* Bench benchmarks [9, 44, 51].”
• MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark (2024, background): “…al., 2023), SEED (Li et al., 2024a), MMBench (Liu et al., 2023d),CV-Bench (Tong et al., 2024a), MM-Vet (Yu et al., 2024), Mantis (Jiang et al., 2024), and BLINK (Fu et al., 2024) have emerged, covering aspects from basic perception to hallucination detection (Cui et al., 2023; Liu et al., 2023a).”
• Depth Anything V2 (2024, unknown): “Lastly, we believe it is also a potential testbed for the 3D awareness of future multimodal LLMs [41, 21, 3].”
• Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs (2025, unknown): “For multi-image/video vision-language benchmarks, we used one multi-image benchmark (BLINK [FHL + 24]) and one video benchmark (VideoMME [FDL + 24]).”
• Qwen3-VL Technical Report (2025, unknown): “We collect diverse real-world multimodal documents sourced from recent Chinese and English websites (Laurençon et al., 2023; Zhu et al., 2023; Li et al., 2024c).”
• VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding (2025, unknown): “To assess VideoLLaMA3’s ability to understand and reason about multiple images in conjunction, we evaluate the model on several widely used benchmarks, including MMMU-Pro [117], MMMU [118], and BLINK [119].”
• LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models (2024, methodology): “…focus on a specific domain for evaluation, including NLVR2 [50] for daily-life VQA, MMMU [61] for colleague-level problem-solving, MathVerse-mv [65] and SciVerse-mv [13] for mathematical and scientific reasoning, BLINK [10] to challenge LMMs, and Mantis-Eval [19] for multi-image understanding.”
• MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark (2024, unknown): “This can be done through more training data (Liu et al., 2023a), better architectures (Ghosh et al., 2024c) or other methods (Fu et al., 2024b).”
• GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning (2025, unknown): “1 [30], MMStar [7], BLINK(val) [11], MUIRBENCH [53], ZeroBench(val) [39], HallusionBench [15], GeoBench [2]; • STEM : MMMU(val) [67], MMMU Pro [68], MathVista [32], MathVision [55], MathVerse [70], DynaMath [74], LogicVista [61], WeMath [37], AI2D [26]; • OCR, Chart & Document : OCRBench [31],…”
• InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models (2025, unknown): “we evaluate the multi-image relation perception and understanding capabilities of InternVL3 across a suite of widely recognized benchmarks, including BLINK [39], Mantis-Eval [51], MMIU [95], MuirBench [118], MMT-Bench [137], and MIRB [153], as presented in Table 4.”
• mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models (2024, background): “BLINK (Fu et al., 2024b) and Q-Bench2 (Zhang et al., 2024d) test the model’s multi-image question answering ability based on low-level visual perception.”