From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions

By Peter Young, Alice Lai, Micah Hodosh, Julia Hockenmaier. 2014

3,236 citations recorded by the source corpus.

We propose to use the visual denotations of linguistic expressions (i.e. the set of images they describe) to define novel denotational similarity metrics, which we show to be at least as beneficial as distributional similarities for two tasks that require semantic inference. To compute these denotational similarities, we construct a denotation graph, i.e. a subsumption hierarchy over constituents and their denotations, based on a large corpus of 30K images and 150K descriptive captions.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation (2017, methodology): “…data in ongoing research. cross-lingual tracks explores data from the WMT 2014 quality estimation task (Bojar et al., 2014).3 Sentences pairs in SNLI derive from Flickr30k image captions (Young et al., 2014) and are labeled with the entailment relations: entailment, neutral, and contradiction.”
A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference (2017, methodology): “SNLI consists only of sentences derived from image captions from the Flickr30k corpus (Young et al., 2014), and thus can be treated as a large additional CAPTIONS genre.”
A large annotated corpus for learning natural language inference (2015, methodology): “For the premises, we used captions from the Flickr30k corpus (Young et al., 2014), a collection of approximately 160k captions (corresponding to about 30k images) collected in an earlier crowd-sourced effort.”
ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks (2019, methodology): “In this ‘zero-shot’ task, we directly apply the pretrained the multi-modal alignment prediction mechanism to caption-based image retrieval in Flickr30k [26] without ﬁne-tuning (thus the description as ‘zero-shot’).”
TAP: Text-Aware Pre-training for Text-VQA and Text-Caption (2020, methodology): “, image-text retrieval [60], visual question answering [6], visual grounding [28, 58], etc.”
ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph (2020, methodology): “Figure 1: Scenes with subtle differences from Flickr30K dataset [17].”
VirTex: Learning Visual Representations from Textual Annotations (2020, background): “Vision-and-Language Pretraining attempts to learn joint representations of image-text paired data that can be transferred to multimodal downstream tasks such as visual question answering [13, 14, 59, 60], visual reasoning [61, 62], referring expressions [63], and language-based image retrieval [35].”
The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes (2020, background): “In the past few years there has been a surge of interest in multimodal problems, from image captioning [8, 43, 88, 28, 68] to visual question answering (VQA) [3, 26, 27, 33, 71] and beyond.”
ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data (2020, methodology): “We compare our model with state-of-the-art methods on image-retrieval and text-retrieval tasks in different settings on MSCOCO[5] and Flickr30k[6] datasets.”
Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training (2019, methodology): “2015) and Flicker30K (Young et al. 2014), comparing to a bunch of strong baselines.”
Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning (2018, methodology): “…COCO-trained models, out-of-domain for Conceptual-trained models); the Conceptual test set (out-of-domain for COCO-trained models, in-domain for Conceptual-trained models); and the Flickr (Young et al., 2014) 1K test set (out-of-domain for both COCO-trained models and Conceptual-trained models).”
Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models (2014, methodology): “We perform the same experimental procedure as done by [15] on the Flickr8K [3] and Flickr30K [42] datasets.”
Deep Fragment Embeddings for Bidirectional Image Sentence Mapping (2014, methodology): “In particular, we report dramatic improvements over state of the art methods on image-sentence retrieval tasks on Pascal1K [2], Flickr8K [3] and Flickr30K [4] datasets.”
Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics (2013, methodology): “We have released a larger dataset (Flickr30K) containing over 30,000 images [Young et al., 2014].”