Unsupervised Visual Sense Disambiguation for Verbs using Multimodal Embeddings

By Spandana Gella, Mirella Lapata, Frank Keller, Frank Keller. 2016

54 citations recorded by the source corpus.

We introduce a new task, visual sense disambiguation for verbs: given an image and a verb, assign the correct sense of the verb, i.e., the one that describes the action depicted in the image. Just as textual word sense disambiguation is useful for a wide range of NLP tasks, visual sense disambiguation can be useful for multimodal tasks such as image retrieval, image description, and text illustration. We introduce VerSe, a new dataset that augments existing multimodal datasets (COCO and TUHOI) with sense labels. We propose an unsupervised algorithm based on Lesk which performs visual sense disambiguation using textual, visual, or multimodal embeddings. We find that textual embeddings perform well when gold-standard textual annotations (object labels and image descriptions) are available, while multimodal embeddings perform well on unannotated images. We also verify our findings by using the textual and multimodal embeddings as features in a supervised setting and analyse the performance of visual sense disambiguation task. VerSe is made publicly available and can be downloaded at: this https URL

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Mohit Bansal
2. Devi Parikh
3. Jianfeng Gao
4. Desmond Elliott
5. Miltiadis Allamanis
6. Lucia Specia
7. Charles Sutton
8. Dhruv Batra

In Computer Vision and Pattern Recognition

1. Mohit Bansal
2. Desmond Elliott
3. Devi Parikh
4. Dhruv Batra
5. Luc Van Gool
6. Trevor Darrell
7. Dacheng Tao
8. Xiaogang Wang

In Natural Language Processing

1. Charles Sutton
2. Premkumar T. Devanbu
3. Miltiadis Allamanis
4. Earl T. Barr
5. Desmond Elliott
6. Lucia Specia
7. Loïc Barrault
8. Dhruv Batra

In Multimodal Learning

1. Desmond Elliott
2. Loïc Barrault
3. Lucia Specia
4. Dhruv Batra
5. Fethi Bougares
6. Devi Parikh
7. Mohit Bansal
8. Stella Frank

Citations in the corpus

• A Survey of Machine Learning for Big Code and Naturalness (2017, background): “ code. Learning to exploit multiple views simultaneously can help machine learning models generalize and tackle issues with data sparsity. Multi-view [193] and multi-modal learning (e.g. Gella et al. [70]), areas actively explored in machine learning, aim to achieve exactly this. By combining multip”
• Resolving Language and Vision Ambiguities Together: Joint Segmentation & Prepositional Attachment Resolution in Captioned Scenes (2016, background): “In a more recent work, [23] studied the problem of reasoning about an image and a verb, where they attempt to pick the correct sense of the verb that describes the action depicted in the image.”
• Findings of the Second Shared Task on Multimodal Machine Translation and Multilingual Image Description (2017, methodology): “We based our selection on the VerSe dataset (Gella et al., 2016), which annotates a subset of the COCO (Lin et al.”