Hallusionbench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models

By Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob and 2 more. Computer Vision and Pattern Recognition, 2023

618 citations recorded by the source corpus.

We introduce “HALLUSIONBENCH11“Hallusion” is a portmanteau of “hallucination” and “illusion.”,” a comprehensive benchmark designed for the evaluation of image-context rea-soning. This benchmark presents significant challenges to advanced large visual-language models (LVLMs), such as GPT-4V(ision), Gemini Pro Vision, Claude 3, and LLaVA-1.5, by emphasizing nuanced understanding and interpre-tation of visual data. The benchmark comprises 346 images paired with 1129 questions, all meticulously crafted by human experts. We introduce a novel structure for these visual questions designed to establish control groups. This structure enables us to conduct a quantitative analysis of the models' response tendencies, logical consistency, and various failure modes. In our evaluation on Hallusion-bench, we benchmarked 15 different models, highlighting a 31.42% question-pair accuracy achieved by the state-of-the-art GPT-4V. Notably, all other evaluated models achieve accuracy below 16%. Moreover, our analysis not only high-lights the observed failure modes, including language hal-lucination and visual illusion but also deepens an under-standing of these pitfalls. Our comprehensive case studies within Hallusionbench shed light on the challenges of hallucination and illusion in LVLMs. Based on these in-sights, we suggest potential pathways for their future im-provement. The benchmark and codebase can be accessed at https://github.com/tianyi-labIHallusionBench.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/0b395ed1c8b284e551172b728e83cf257e33729a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Tong Lu
2. Luc Van Gool
3. Xizhou Zhu
4. Dahua Lin
5. Dacheng Tao
6. Xiaogang Wang
7. Chunhua Shen
8. Chen Change Loy

Citations in the corpus

• Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Models (2024, background): “This includes efforts aimed at evaluating the extent and nature of hallucination across different models and scenarios [120, 107, 121, 122].”
• GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning (2025, neutral): “1 [30], MMStar [7], BLINK(val) [11], MUIRBENCH [53], ZeroBench(val) [39], HallusionBench [15], GeoBench [2]; • STEM : MMMU(val) [67], MMMU Pro [68], MathVista [32], MathVision [55], MathVerse [70], DynaMath [74], LogicVista [61], WeMath [37], AI2D [26]; • OCR, Chart & Document : OCRBench [31],…”
• LlaVA-CoT: Let Vision Language Models Reason Step-By-Step (2024, neutral): “…on several multimodal reasoning benchmarks, including MMStar [10], MMBench [35], MMVet [64], MathVista [37], AI2D [25], and Hal-lusionBench [18], and observed that LLaVA-CoT offers two primary advantages: First, enabling the model to perform structured reasoning independently substantially…”
• InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models (2025, neutral): “We evaluate InternVL’s propensity for hallucinations on four established benchmarks—HallusionBench [45], MMHal-Bench [111], CRPE [126], and POPE [67]—as detailed in Table 5.”
• Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (2024, neutral): “We evaluate InternVL’s tendency toward hallucinations across four different benchmarks, including: HallusionBench [77]: HallusionBench is a benchmark for evaluating image-context reasoning in MLLMs through a Yes/No judgment question format, focusing on challenges such as language hallucination and…”
• InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model (2024, neutral): “We report results in Math-Vista [52], MMMU [91], AI2D [40], MME [27], MM-Bench [51], MMBench-Chinese [51], SEED-Bench (Image) [41], LLaVA-Bench (In-the-Wild) [49], QBench [85], MM-Vet [90], HallusionBench [31], ChartQA [56], and POPE [45].”
• InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency (2025, neutral): “1 (en) [71], MMStar [11], BLINK [36], HallusionBench [41], AI2D [55], OCRBench [72], MMVet [168], MME-RealWorld (en) [178], MVBench [63], VideoMME [35], MMMU [170], MathVista [76], MathVision [134], MathVerse [175], DynaMath [189], WeMath [100], Log-icVista [153], MATH500 [45], AIME24 [84], AIME25…”
• R1-VL: Learning to Reason with Multimodal Large Language Models via Step-Wise Group Relative Policy Optimization (2025, positive): “For evaluation, we adopt 8 widely-used multimodal benchmarks for comprehensively evaluating our proposed StepGRPO, including MathVista [21], MM-Star [5], Math-Vision [36], ChartQA [24], DynaMath [51], HallusionBench [11], MathVerse [48] and MME [10].”
• Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models (2025, neutral): “Besides, we select four general multimodal benchmarks to demenstrate the general ability of our model: MM-Star [47], ChartQA [48], MME [49] and HallBench [50].”
• Mini-Gemini: Mining the Potential of Multi-Modality Vision Language Models (2024, neutral): “Instruction :Capture a playful child pre-tending to talk on the phone.”
• How far are we to GPT-4V? Closing the gap to commercial multimodal models with open-source suites (2024, positive): “Specifically, our model achieves the best performance on HallusionBench [30], demonstrating its outstanding ability to reduce hallucinations.”
• Qwen3-Omni Technical Report (2025, neutral): “To assess performance in general visual question answering, the model is evaluated on MMStar (Chen et al., 2024a), HallusionBench (Guan et al., 2024), and MM-MT-Bench (Agrawal et al., 2024).”