Let there be a clock on the beach: Reducing Object Hallucination in Image Captioning

By Ali Furkan Biten, Lluis Gomez i Bigorda, Dimosthenis Karatzas. 2021

98 citations recorded by the source corpus.

Explaining an image with missing or non-existent objects is known as object bias (hallucination) in image captioning. This behaviour is quite common in the state-of-the-art captioning models which is not desirable by humans. To decrease the object hallucination in captioning, we propose three simple yet efficient training augmentation method for sentences which requires no new training data or increase in the model size. By extensive analysis, we show that the proposed methods can significantly diminish our models’ object bias on hallucination metrics. Moreover, we experimentally demonstrate that our methods decrease the dependency on the visual features. All of our code, configuration files and model weights are available online1.

Subfields: Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Survey of Hallucination in Natural Language Generation (2022, background): “[13] hypothesize that the main cause of object hallucination is the systematic co-occurence of particular object categories in input images.”
Plausible May Not Be Faithful: Probing Object Hallucination in Vision-Language Pre-training (2022, background): “, in object detection or image captioning) and is called object hallucination (Rohrbach et al., 2018; Biten et al., 2022).”
Evaluating Object Hallucination in Large Vision-Language Models (2023, background): “Especially, LLMs tend to hallucinate unintended text (Huang et al., 2021; Bang et al., 2023), and VLPMs might generate nonexistent objects in the image (Biten et al., 2022) (termed as object hal-lucination ).”
Analyzing and Mitigating Object Hallucination in Large Vision-Language Models (2023, methodology): “, 2021), ROI feature fusion (Biten et al., 2022), and eliminating co-occurrence patterns through data augmentation (Kim et al.”
Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding (2023, methodology): “In the realm of VLMs, “object hal-lucination” is also well-documented, referring to models producing plausible outputs that include objects that do not match or are missing from images [6, 38, 54].”
Aligning Large Multimodal Models with Factually Augmented RLHF (2023, result): “…the absence of a negative feedback mechanism preventing the model from identifying hallucinations in its output, and the potential limitations of our Fact-RLHF reward model, especially when compared against the high-capacity oracle models in previous successful studies (Touvron et al., 2023b).”