Probing Text Models for Common Ground with Visual Representations

By Gabriel Ilharco, Rowan Zellers, Ali Farhadi, Hannaneh Hajishirzi. 2020

14 citations recorded by the source corpus.

Vision, as a central component of human perception, plays a fundamental role in shaping natural language. To better understand how text models are connected to our visual perceptions, we propose a method for examining the similarities between neural representations extracted from words in text and objects in images. Our approach uses a lightweight probing model that learns to map language representations of concrete words to the visual domain. We find that representations from models trained on purely textual data, such as BERT, can be nontrivially mapped to those of a vision model. Such mappings generalize to object categories that were never seen by the probe during training, unlike mappings learned from permuted or random representations. Moreover, we find that the context surrounding objects in sentences greatly impacts performance. Finally, we show that humans significantly outperform all examined models, suggesting considerable room for improvement in representation learning and grounding.

Subfields: Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Multimodal Learning

Citations in the corpus

Pre-Trained Language Models for Interactive Decision-Making (2022, neutral): “The possibility that LMs might encode non-linguistic information useful for other downstream tasks is suggested by a number of recent studies [34, 16, 21].”
Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents (2022, background): “It has been shown by recent works that large language models (LLMs) pre-trained on large unstructured text corpus not only can perform strongly on various down-stream NLP tasks [10, 39, 40, 5] but the learned representations can also be used to model relations of entities [23], retrieve matching vis”
X-LXMERT: Paint, Caption and Answer Questions with Multi-Modal Transformers (2020, background): “These impressive results as well as recent probing mechanisms (Ilharco et al., 2020) suggest that these models are able to capture a variety of semantics in images including objects, attributes and their relationships and ground these in natural language.”
Contrastive Representation Learning: A Framework and Review (2020, methodology): “[50] learned a probe to find the similarities between words and object images from a paired image captioning dataset.”
Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Color (2021, methodology): “Finally, Ilharco et al. (2020) train a probe to map LM representations of textual captions to paired visual representations of image patches, in order to evaluate how useful the former are for discerning between different visual representations.”
Seeing past words: Testing the cross-modal capabilities of pretrained V&L models on counting tasks (2020, background): “Ilharco et al. (2020) show that contextual text-only language models such as BERT encode visual representations reasonably well, though they fall short of human performance.”
Unsupervised Vision-and-Language Pre-training Without Parallel Images and Captions (2021, result): “This also holds for the other two models and resembles what is observed in Li et al. (2020b) and Ilharco et al. (2020).”