Probing Contextual Language Models for Common Ground with Visual Representations

By Gabriel Ilharco, Rowan Zellers, Ali Farhadi, Hannaneh Hajishirzi. 2020

28 citations recorded by the source corpus.

The success of large-scale contextual language models has attracted great interest in probing what is encoded in their representations. In this work, we consider a new question: to what extent contextual representations of concrete nouns are aligned with corresponding visual representations? We design a probing model that evaluates how effective are text-only representations in distinguishing between matching and non-matching visual representations. Our findings show that language representations alone provide a strong signal for retrieving image patches from the correct object categories. Moreover, they are effective in retrieving specific instances of image patches; textual context plays an important role in this process. Visually grounded language models slightly outperform text-only language models in instance retrieval, but greatly under-perform humans. We hope our analyses inspire future research in understanding and improving the visual capabilities of language models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

Contrastive Learning of Medical Visual Representations from Paired Images and Text (2020, background): “Among existing studies, Ilharco et al. (2021) and Gupta et al. (2020) explored cross-modality contrastive objectives related to ours, but for the purpose of probing visual-linguistic models and learning phrase grounding, respectively.”
Implicit Representations of Meaning in Neural Language Models (2021, methodology): “…deal of past research uses language modeling as a pretraining scheme for more conventional (supervised) semantics tasks in NLP. LM pretraining is useful for semantic parsing (Einolghozati et al., 2019), instruction following (Hill et al., 2020), and even image retrieval (Ilharco et al., 2021).”
Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks (2023, background): “…being trained on only textual data, LMs have been shown to be able to structure their representations of perceptual concepts such as size and color (Abdou et al., 2021; Patel and Pavlick, 2022; Zhang et al., 2020; Ilharco et al., 2021; i.a. ) in a way that credibly mirrors the physical world.”
Seeing past words: Testing the cross-modal capabilities of pretrained V&L models on counting tasks (2020, background): “Ilharco et al. (2020) show that contextual text-only language models such as BERT encode visual representations reasonably well, though they fall short of human performance.”
Generating Images with Multimodal Language Models (2023, background): “Prior work showcased the effectiveness of grounding text-only LMs to images for vision-and-language tasks [56, 4, 29, 33, 31, 35], to embodied settings for robotics [3, 18], offline reinforcement learning [48], and more.”