FOIL it! Find One mismatch between Image and Language caption

By Ravi Shekhar, Sandro Pezzelle, Yauhen Klimovich, Aurélie Herbelot, Moin Nabi, Enver Sangineto, Raffaella Bernardi. 2017

162 citations recorded by the source corpus.

In this paper, we aim to understand whether current language and vision (LaVi) models truly grasp the interaction between the two modalities. To this end, we propose an extension of the MS-COCO dataset, FOIL-COCO, which associates images with both correct and ‘foil’ captions, that is, descriptions of the image that are highly similar to the original ones, but contain one single mistake (‘foil word’). We show that current LaVi models fall into the traps of this data and perform badly on three tasks: a) caption classification (correct vs. foil); b) foil word detection; c) foil word correction. Humans, in contrast, have near-perfect performance on those tasks. We demonstrate that merely utilising language cues is not enough to model FOIL-COCO and that it challenges the state-of-the-art by requiring a fine-grained understanding of the relation between text and image.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Evaluation & Benchmarks

Citations in the corpus

Targeted Syntactic Evaluation of Language Models (2018, background): “…(Rimell et al., 2009; Nivre et al., 2010; Bender et al., 2011), and minimal pair approaches have been proposed for evaluating image captioning (Shekhar et al., 2017) and machine translation systems (Sennrich, 2017), but no data sets exist that target a range of syntactic constructions for…”
Evaluating Models’ Local Decision Boundaries via Contrast Sets (2020, background): “Challenge sets exist for various phenomena, including ones with “minimal” edits similar to our contrast sets, e.g., in image captioning (Shekhar et al., 2017), machine translation (Sen-nrich, 2017; Burlot and Yvon, 2017; Burlot et al., 2018), and language modeling (Marvin and Linzen, 2018; Warstadt…”
Evaluating NLP Models via Contrast Sets (2020, background): “Challenge sets exist for various phenomena, including ones with “minimal” edits similar to our contrast sets, e.g., in image captioning (Shekhar et al., 2017), machine translation (Sennrich, 2017), and language modeling (Marvin and Linzen, 2018; Warstadt et al., 2019).”
Grounding Visual Explanations (2018, methodology): “Also following [24], for the third task we replace the foiled word with words from a set of target words and choose a target word based on which one maximizes the score of the classifier.”
Attacking Visual Language Grounding with Adversarial Examples: A Case Study on Neural Image Captioning (2017, background): “The recent work in (Shekhar et al., 2017) touched upon the robustness of neural image cap-tioning for language grounding by showing its in-sensitivity to one-word (foil word) changes in the language caption, which corresponds to the untar-geted attack category in adversarial examples.”
Competency Problems: On Finding and Removing Artifacts in Language Data (2021, methodology): “Many works have tried to remove data artifacts by making minimal changes to existing data (Shekhar et al., 2017; Sennrich, 2017; Zhao et al., 2018, inter alia ).”
Seeing past words: Testing the cross-modal capabilities of pretrained V&L models on counting tasks (2020, background): “Matching imagesentence pairs has featured in previous work on foiling, albeit in the context of specific tasks with relatively small datasets (Shekhar et al., 2017).”
Unified Visual-Semantic Embeddings: Bridging Vision and Language With Structured Meaning Representations (2019, background): “Recent works [40, 39] have raised their concerns on the robustness of the learned visual-semantic embeddings.”
Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality (2022, background): “…stress tests about correctly understanding implausible scenes [13], exploitation of language and vision priors [11, 27], single word mismatches [64], hate speech detection [26, 32, 41, 92], memes [39, 75], ablation of one modality to probe the other [22], distracting models with visual…”
Adversarial Evaluation of Multimodal Machine Translation (2018, background): “…is related to the foiled image captions evaluation, in which the performance of an image captioning system is measured when a single word is replaced with an incorrect, but similar word (Shekhar et al., 2017); the main difference is that we replace the visual data instead of manipulating the text.”
Challenges and Prospects in Vision and Language Research (2019, background): “Here the algorithm, must determine if the caption has been FOILed and then detect the FOIL word and replace it with a correct word.”