Illustrative Language Understanding: Large-Scale Visual Grounding with Image Search

By Jamie Ryan Kiros, William Chan, Geoffrey E. Hinton. 2018

54 citations recorded by the source corpus.

We introduce Picturebook, a large-scale lookup operation to ground language via ‘snapshots’ of our physical world accessed through image search. For each word in a vocabulary, we extract the top-k images from Google image search and feed the images through a convolutional network to extract a word embedding. We introduce a multimodal gating function to fuse our Picturebook embeddings with other word representations. We also introduce Inverse Picturebook, a mechanism to map a Picturebook embedding back into words. We experiment and report results across a wide range of tasks: word similarity, natural language inference, semantic relatedness, sentiment/topic classification, image-sentence ranking and machine translation. We also show that gate activations corresponding to Picturebook embeddings are highly correlated to human judgments of concreteness ratings.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Retrieval & Search, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jason Baldridge
2. Yinfei Yang
3. Daniel Matthew Cer
4. Zarana Parekh
5. Gabriel Ilharco
6. Hung Le
7. Steven C. H. Hoi
8. Austin Waters

In Computer Vision and Pattern Recognition

1. Jason Baldridge
2. Yinfei Yang
3. Steven C. H. Hoi
4. Doyen Sahoo
5. Zarana Parekh
6. Daniel Matthew Cer
7. Yuan Zhang
8. Hung Le

In Information Retrieval & Search

1. Yinfei Yang
2. Daniel Matthew Cer
3. Jason Baldridge
4. Steven C. H. Hoi

In Multimodal Learning

1. Jason Baldridge
2. Yinfei Yang
3. Daniel Matthew Cer
4. Yuan Zhang
5. Steven C. H. Hoi
6. Hung Le
7. Gabriel Ilharco
8. Nancy F. Chen

Citations in the corpus

• Crisscrossed Captions: Extended Intramodal and Intermodal Semantic Similarity Judgments for MS-COCO (2020, methodology): “improvements for both word similarity ranking and image-text retrieval (Kiros et al., 2018), and querybased training of image models demonstrates language’s power to improve image representations (Juan et al.”
• Large-Scale Representation Learning from Visually Grounded Untranscribed Speech (2019, background): “To address problems of this sort, there are likely many opportunities to combine ideas from unsupervised term discovery (Kamper et al., 2016; Bansal et al., 2017) with audiovisual word learning (Harwath et al., 2018) and models of visual grounding that have been applied to text (Kiros et al., 2018).”
• Multimodal Transformer Networks for End-to-End Video-Grounded Dialogue Systems (2019, methodology): “, 2018) or image-grounded word embedding (Kiros et al., 2018) to improve the semantic understanding capability of the models.”