Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding

By Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, Li Bing. Computer Vision and Pattern Recognition, 2023

737 citations recorded by the source corpus.

Large Vision-Language Models (LVLMs) have advanced considerably, intertwining visual recognition and language understanding to generate content that is not only coherent but also contextually attuned. Despite their success, LVLMs still suffer from the issue of object hallucinations, where models generate plausible yet incorrect outputs that include objects that do not exist in the images. To mitigate this issue, we introduce Visual Contrastive Decoding (VCD), a simple and training-free method that contrasts output distributions derived from original and distorted visual inputs. The proposed VCD effectively reduces the over-reliance on statistical bias and unimodal priors, two essential causes of object hallucinations. This adjustment ensures the generated content is closely grounded to visual inputs, resulting in contextually accurate outputs. Our experiments show that VCD, without either additional training or the usage of external tools, significantly mitigates the object hallucination issue across different LVLM families. Beyond mitigating object hallucinations, VCD also excels in general LVLM benchmarks, highlighting its wide-ranging applicability.

View Paper: https://www.semanticscholar.org/paper/328eb183007bf4aefbf42437b42a15db375803e3

Citations in the corpus

Hallucination of Multimodal Large Language Models: A Survey (2024, unknown): “When trained with such detailed data, in an attempt to fit the detail level and length distribution of ground truth captions, the model may risk expressing details that it cannot discern from the image, and therefore exhibit hallucinations.”
A survey on multimodal large language models (2023, unknown): “• The object hallucination issue is widespread [13, 44], which largely affects the reliability of MLLMs.”
Foundation Models Defining a New Era in Vision: A Survey and Outlook (2025, unknown): “…hallucinations using various methods such as VQAs, classiﬁcation of objects etc; 2) construction of ﬁne-tuning data aimed at reducing object hallucinations [184], [187], and algorithmic approaches to minimize hallucinations post-training [188], [189], [190].”