Seeing past words: Testing the cross-modal capabilities of pretrained V&L models on counting tasks

By Letitia Parcalabescu, Albert Gatt, Anette Frank, Iacer Calixto. 2020

53 citations recorded by the source corpus.

We investigate the reasoning ability of pretrained vision and language (V&L) models in two tasks that require multimodal integration: (1) discriminating a correct image-sentence pair from an incorrect one, and (2) counting entities in an image. We evaluate three pretrained V&L models on these tasks: ViLBERT, ViLBERT 12-in-1 and LXMERT, in zero-shot and finetuned settings. Our results show that models solve task (1) very well, as expected, since all models are pretrained on task (1). However, none of the pretrained V&L models is able to adequately solve task (2), our counting probe, and they cannot generalise to out-of-distribution quantities. We propose a number of explanations for these findings: LXMERT (and to some extent ViLBERT 12-in-1) show some evidence of catastrophic forgetting on task (1). Concerning our results on the counting probe, we find evidence that all models are impacted by dataset bias, and also fail to individuate entities in the visual input. While a selling point of pretrained V&L models is their ability to solve complex tasks, our findings suggest that understanding their reasoning and grounding capabilities requires more targeted investigations on specific phenomena.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

When and why vision-language models behave like bags-of-words, and what to do about it? (2022, background): “Parcalabescu et al. (2021b) show that VLMs have difﬁculties in counting objects in images.”
What's "up" with vision-language models? Investigating their struggle with spatial reasoning (2023, background): “…evaluating more specific phenomena have been proposed, testing understanding of word order (Thrush et al., 2022; Yuksekgonul et al., 2023), counting (Parcalabescu et al., 2021), object-attribute association (Yamada et al., 2022), and compositionality (Kamath et al., 2023; Ma et al., 2022).”
Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality (2022, background): “…similarity between many suitable captions [1,17], collecting more diverse image-caption pairs beyond the predominately English and North American/Western European datasets [50], probing for an understanding of verb-argument relationships [30], counting [53], or speciﬁc model failure modes [65, 69].”