Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality

By Tristan Thrush, Ryan Jiang, Max Bartolo, Adina Williams, Douwe Kiela, Candace Ross. Computer Vision and Pattern Recognition, 2022

639 citations recorded by the source corpus.

We present a novel task and dataset for evaluating the ability of vision and language models to conduct visio-linguistic compositional reasoning, which we call Winoground. Given two images and two captions, the goal is to match them correctly-but crucially, both captions contain a completely identical set of words, only in a different order. The dataset was carefully hand-curated by expert annotators and is labeled with a rich set offine-grained tags to assist in analyzing model performance. We probe a diverse range of state-of-the-art vision and language models and find that, surprisingly, none of them do much better than chance. Evidently, these models are not as skilled at visio-linguistic compositional reasoning as we might have hoped. We perform an extensive analysis to obtain insights into how future work might try to mitigate these models' shortcomings. We aim for Winoground to serve as a useful evaluation set for advancing the state of the art and driving further progress in the field. The dataset is available at https://huggingface.co/datasets/facebook/winoground.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/c435ecd0321dcec1f25e458bf930311f9e1d04b6

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis (2022, background): “Previous work also suggests the deficiency of large pre-trained models in compositional understanding (Thrush et al., 2022).”
Evaluating Text-to-Visual Generation with Image-to-Text Generation (2024, methodology): “However, CLIP cannot reliably process compositional text prompts [30, 44, 75, 88].”
MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning (2023, methodology): “Visual Reasoning includes the Winoground (Thrush et al., 2022b), VSR (Liu et al., 2022) and IconQA (Lu et al., 2021) dataset.”
A very preliminary analysis of DALL-E 2 (2022, positive): “Similarly, Thrush et al. (2022) tested the visuo-linguistic compositional reasoning of current state-of-the-art vision and language models using the Winoground benchmark.”
What's "up" with vision-language models? Investigating their struggle with spatial reasoning (2023, background): “More recently, vision-language benchmarks evaluating more specific phenomena have been proposed, testing understanding of word order (Thrush et al., 2022; Yuksekgonul et al., 2023), counting (Parcalabescu et al., 2021), object-attribute association (Yamada et al., 2022), and compositionality…”
Your Diffusion Model is Secretly a Zero-Shot Classifier (2023, positive): “Our approach also outperforms SOTA contrastive methods on the challenging Winoground compositional reasoning benchmark [74].”
SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities (2024, positive): “Many prior works have focused on benchmarking VLMs [58, 66], considering tasks like VQA (e.g. VQAv2 [28], OK-VQA [47], COCO [41], or Visual Genome [37]).”
When and why vision-language models behave like bags-of-words, and what to do about it? (2022, background): “Our main goal is to propose an isolated ﬁne-grained evaluation of relation and attribution understanding with a large scale dataset; hence we believe our dataset complements Winoground.”
Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs (2024, background): “More recently, Thrush et al. [56], Yuk-sekgonul et al. [65] and Hsieh et al. [19] study the binding problems CLIP faces in processing text queries, noting that CLIP models treat text input as a bag of words.”
Mieb: Massive Image Embedding Benchmark (2025, unknown): “Especially, WinoGround [53] is extremely challenging (see Appendix) due to its image and textual confounders.”
Better Language Models Exhibit Higher Visual Alignment (2024, unknown): “By integrating off-the-shelf LLMs into a lightweight CLIP-like architecture, we leverage their large-scale pre-training, intrinsic knowledge of the visual world, and multilingual capabilities, achieving competitive performance with conventional VLMs trained on significantly larger datasets.”