Detecting and Preventing Hallucinations in Large Vision Language Models

By Anisha Gunjal, Jihan Yin, Erhan Bas. AAAI Conference on Artificial Intelligence, 2023

337 citations recorded by the source corpus.

Instruction tuned Large Vision Language Models (LVLMs) have significantly advanced in generalizing across a diverse set of multi-modal tasks, especially for Visual Question Answering (VQA). However, generating detailed responses that are visually grounded is still a challenging task for these models. We find that even the current state-of-the-art LVLMs (InstructBLIP) still contain a staggering 30 percent of the hallucinatory text in the form of non-existent objects, unfaithful descriptions, and inaccurate relationships. To address this, we introduce M-HalDetect, a Multimodal Hallucination Detection Dataset that can be used to train and benchmark models for hallucination detection and prevention. M-HalDetect consists of 16k fine-grained annotations on VQA examples, making it the first comprehensive multi-modal hallucination detection dataset for detailed image descriptions. Unlike previous work that only consider object hallucination, we additionally annotate both entity descriptions and relationships that are unfaithful. To demonstrate the potential of this dataset for hallucination prevention, we optimize InstructBLIP through our novel Fine-grained Direct Preference Optimization (FDPO). We also train fine-grained multi-modal reward models from InstructBLIP and evaluate their effectiveness with best-of-n rejection sampling (RS). We perform human evaluation on both FDPO and rejection sampling, and find that they reduce hallucination rates in InstructBLIP by 41% and 55% respectively. We also find that our reward model generalizes to other multi-modal models, reducing hallucinations in LLaVA and mPLUG-OWL by 15% and 57% respectively, and has strong correlation with human evaluated accuracy scores. The dataset is available at https://github.com/hendryx-scale/mhal-detect.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Sociology and Political Science

View Paper: https://www.semanticscholar.org/paper/658cd67a91da86cf451e6f1b015f762b56015172

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Sociology and Political Science

Citations in the corpus

DreamLLM: Synergistic Multimodal Comprehension and Creation (2023, background): “…hallucination, a phenomenon where MLLMs generate non-existent objects or identities in images, significantly compromises their multimodal comprehension capabilities (Dai et al., 2023b; Liu et al., 2023a; Gunjal et al., 2023) and may pose safety risks (MacLeod et al., 2017; Rohrbach et al., 2018).”
Foundation Models Defining a New Era in Vision: A Survey and Outlook (2025, neutral): “…hallucinations using various methods such as VQAs, classiﬁcation of objects etc; 2) construction of ﬁne-tuning data aimed at reducing object hallucinations [184], [187], and algorithmic approaches to minimize hallucinations post-training [188], [189], [190].”
A Survey on Hallucination in Large Vision-Language Models (2024, neutral): “One notable dataset is M-HalDetect [Gunjal et al. , 2023].”
Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding (2023, background): “Several recent studies address this issue by proposing hallucination-targeted datasets for fine-tuning [20, 42], training a post-hoc revisor to reconstruct less hallucinatory outputs [77] or adapting factually augmented Reinforcement Learning from Human Feedback (RLHF) [60].”
A Survey of Hallucination in Large Foundation Models (2023, methodology): “To tackle this issue, the study (Gunjal et al., 2023)introduces MHalDetect1, a Multimodal Hallucination Detection Dataset designed for training and evaluating models aimed at detecting and preventing hallucinations.”
Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models (2023, positive): “Gunjal et al. (2023) introduce a multi-modal hallucination detection dataset named M-HalDetect, further study the unfaithful descriptions and inaccurate relationships beyond object hallucinations in LVLMs.”
Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning (2024, background): “While these models exhibit impressive generation capabilities, they suffer from inaccurate information [30] and even hallucinations [15].”
RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-Grained Correctional Human Feedback (2023, background): “[15] distinguishes the inaccurate parts in responses via human annotation, and internally discourages the halluci-nated parts by direct preference optimization.”
Hallusionbench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models (2023, background): “…also the first work to include human-edited images to assess the robustness of current LVLMs. Additionally, unlike existing benchmarks, H ALLUSION B ENCH focuses on evaluating both language hallucinations and visual illusions, moving beyond the narrow scope of object hallucinations [19, 24, 29].”
Analyzing and Mitigating Object Hallucination in Large Vision-Language Models (2023, background): “…to address the issue of object hallucination in LVLMs, including hallucination evaluation and detection (Wang et al., 2023a; Liu et al., 2023a; Li et al., 2023d), as well as the construction of higher-quality datasets for fine-tuning (Gunjal et al., 2023; Li et al., 2023c; Liu et al., 2023a;d).”