When and why vision-language models behave like bags-of-words, and what to do about it?

By Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, Daniel Jurafsky, James Y. Zou. International Conference on Learning Representations, 2022

670 citations recorded by the source corpus.

Despite the success of large vision and language models (VLMs) in many downstream applications, it is unclear how well they encode compositional information. Here, we create the Attribution, Relation, and Order (ARO) benchmark to systematically evaluate the ability of VLMs to understand different types of relationships, attributes, and order. ARO consists of Visual Genome Attribution, to test the understanding of objects' properties; Visual Genome Relation, to test for relational understanding; and COCO&Flickr30k-Order, to test for order sensitivity. ARO is orders of magnitude larger than previous benchmarks of compositionality, with more than 50,000 test cases. We show where state-of-the-art VLMs have poor relational understanding, can blunder when linking objects to their attributes, and demonstrate a severe lack of order sensitivity. VLMs are predominantly trained and evaluated on large datasets with rich compositional structure in the images and captions. Yet, training on these datasets has not been enough to address the lack of compositional understanding, and evaluating on these datasets has failed to surface this deficiency. To understand why these limitations emerge and are not represented in the standard tests, we zoom into the evaluation and training procedures. We demonstrate that it is possible to perform well on retrieval over existing datasets without using the composition and order information. Given that contrastive pretraining optimizes for retrieval on datasets with similar shortcuts, we hypothesize that this can explain why the models do not need to learn to represent compositional information. This finding suggests a natural solution: composition-aware hard negative mining. We show that a simple-to-implement modification of contrastive learning significantly improves the performance on tasks requiring understanding of order and compositionality.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/10667c1ae4b49808772b5a377c5b52196701267f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark (2024, background): “If models rely on superficial cues rather than true multimodal understanding (Du et al., 2023; Yuksekgonul et al., 2023), we risk overestimating their capabilities and potentially deploying systems that fail in unpredictable ways when faced with novel scenarios (Wu and Xie, 2024; Tong et al.,…”
Evaluating Text-to-Visual Generation with Image-to-Text Generation (2024, background): “Studies [30, 44, 52, 79, 88] show that VLMs like CLIP struggle with compositional text prompts involving multiple objects, attribute bindings, spatial/action relations, counting, and logical reasoning.”
Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs (2024, background): “…from two potential concerns of the current multi-modal learning research: 1) relying too heavily too early on language can act as a shortcut [40, 135], compensating for the deficiencies in learning effective visual representations, and 2) existing benchmarks may not provide adequate guidance…”
Hallucination of Multimodal Large Language Models: A Survey (2024, positive): “The definition of hallucination types aligns well with the domain of compositional generalization [79, 121] of VLMs, which investigates visio-linguistic generalization and reasoning abilities.”
Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs (2024, background): “For example, Tong et al. [57], Yuksekgonul et al. [65] demonstrate that failure patterns in the pretrained text encoder [43, 44] will lead to downstream failures in text-guided generative models [22, 46].”
On Evaluating Adversarial Robustness of Large Vision-Language Models (2023, methodology): “While aligned image and text encoders have been shown to perform well on vision-language tasks [65], recent research suggests that VLMs may behave like bags-of-words [103] and therefore may not be dependable for optimizing cross-modality similarity.”
What's "up" with vision-language models? Investigating their struggle with spatial reasoning (2023, methodology): “…CLIP (Radford et al., 2021) ViT-B/32 and ViT-L/14; a version of CLIP ViT-B/32 that has been finetuned on word order shuffling data called negCLIP (Yuksekgonul et al., 2023); a version of CLIP ViT-B/32 that has been initialized with RoBERTa-pretrained weights (Ilharco et al., 2021); CoCa, a model…”
Nomic Embed Vision: Expanding the Latent Space (2024, unknown): “(Yuksekgonul et al., 2023; Paiss et al., 2023).”
Mieb: Massive Image Embedding Benchmark (2025, unknown): “Commonly, it involves distinguishing a ground truth from hard negatives with perturbed inputs, e.g., word order shuffling in ARO benchmark [66].”