Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs

By Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, Saining Xie. Computer Vision and Pattern Recognition, 2024

815 citations recorded by the source corpus.

Is vision good enough for language? Recent advancements in multimodal models primarily stem from the powerful reasoning abilities of large language models (LLMs). However, the visual component typically depends only on the instance-level contrastive language-image pre-training (CLIP). Our research reveals that the visual capabilities in recent MultiModal LLMs (MLLMs) still exhibit systematic shortcomings. To understand the roots of these errors, we explore the gap between the visual embedding space of CLIP and vision-only self-supervised learning. We identify “CLIP-blind pairs”- images that CLIP perceives as similar despite their clear visual differences. With these pairs, we construct the Multimodal Visual Patterns (MMVP) benchmark. MMVP exposes areas where state-of-the-art systems, including GPT-4V, struggle with straightforward questions across nine basic visual patterns, often providing incorrect answers and hallucinated explanations. We further evaluate various CLIP-based vision-and-language models and found a notable correlation between visual patterns that challenge CLIP models and those problematic for multimodal LLMs. As an initial effort to address these issues, we propose a Mixture of Features (MoF) approach, demonstrating that integrating vision self-supervised learning features with MLLMs can significantly enhance their visual grounding capabilities. Together, our research suggests visual representation learning remains an open challenge, and accurate visual grounding is crucial for future successful multimodal systems.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/ca00f4056f9039d3c1a4c3a113f5ee0527149b66

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

How far are we to GPT-4V? Closing the gap to commercial multimodal models with open-source suites (2024, neutral): “[106] observed notable differences in the visual patterns of CLIP and DINOv2 [84], leading to the development of a mixture-of-features module that combines these two VFMs. LLaVA-HR [72] introduced a dual-branch vision encoder utilizing CLIP-ViT for low-resolution pathways and CLIP-ConvNext for…”
DeepSeek-VL: Towards Real-World Vision-Language Understanding (2024, methodology): “To address these limitations, recent researches (Lin et al., 2023b; Tong et al., 2024; Wei et al., 2023) have advocated for the integration of additional vision-only self-supervised encoders, to enhance the visual grounding capabilities of multi-modality models.”
SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training (2025, neutral): “Prior efforts to enhance VLMs’ visual perception include combining multiple visual encoders (Tong et al., 2024d; Kar et al., 2025; Tong et al., 2024a), curating high-quality SFT data (Chen et al., 2023; Liu et al., 2024; Tong et al., 2024a), and improving the SFT training recipe by unfreezing the…”
Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Models (2024, methodology): “Table 5 and 6 show the statistics of the datasets we used, including IsoBench [8], BLINK [9], MMVP [44], and V ∗ Bench [51].”
MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark (2024, background): “…models rely on superficial cues rather than true multimodal understanding (Du et al., 2023; Yuksekgonul et al., 2023), we risk overestimating their capabilities and potentially deploying systems that fail in unpredictable ways when faced with novel scenarios (Wu and Xie, 2024; Tong et al., 2024b).”
VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks (2024, background): “…in vision-language models (VLMs) which can process image-text inputs to generate text, used for many multimodal tasks such as image captioning (Vinyals et al., 2015), visual question answering (Antol et al., 2015), and other benchmarks (Mialon et al., 2023; Yue et al., 2023; Tong et al., 2024).”
SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models (2024, positive): “To investigate the ability of SPHINX-MoE more concretely and locate its ability level among many developed MLLMs, we evaluate SPHINX-MoE on some recently curated benchmarks, which are listed below: - MMVP (Tong et al., 2024): A benchmark specially crafted to measure MLLMs’ visual understanding capab”
Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs (2024, methodology): “Recent work [80, 117] advocates for incorporating these diverse vision models into MLLMs.”
Emerging Properties in Unified Multimodal Pretraining (2025, neutral): “0 -EN) [46], MM-Vet [96], MMMU [97], MathVista [49], and MMVP [74].”
ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation (2024, background): “However, despite showing promises for open-world planning and goal specification [99–113], the caption-guided pre-training scheme of VLMs often limits visual details that can be retained about the image [114–117].”