Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics

By Micah Hodosh, Peter Young, Julia Hockenmaier. 2013

1,460 citations recorded by the source corpus.

The ability to associate images with natural language sentences that describe what is depicted in them is a hallmark of image understanding, and a prerequisite for applications such as sentence-based image search. In analogy to image search, we propose to frame sentence-based image annotation as the task of ranking a given pool of captions. We introduce a new benchmark collection for sentence-based image description and search, consisting of 8,000 images that are each paired with five different captions which provide clear descriptions of the salient entities and events. We introduce a number of systems that perform quite well on this task, even though they are only based on features that can be obtained with minimal supervision. Our results clearly indicate the importance of training on multiple captions per image, and of capturing syntactic (word order-based) and semantic features of these captions. We also perform an in-depth comparison of human and automatic evaluation metrics for this task, and propose strategies for collecting human judgments cheaply and on a very large scale, allowing us to augment our collection with additional relevance judgments of which captions describe which image. Our analysis shows that metrics that consider the ranked list of results for each query image or sentence are significantly more robust than metrics that are based on a single response per query. Moreover, our study suggests that the evaluation of ranking-based image description systems may be fully automated.

Subfields: Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Supervised Learning of Universal Sentence Representations from Natural Language Inference Data (2017, methodology): “Caption-Image retrieval The caption-image retrieval task evaluates joint image and language feature models (Hodosh et al., 2013; Lin et al., 2014).”
Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books (2015, background): “Related to ours is also work on image retrieval [11] which aims to find an image that best depicts a complex description.”
Learning Transferable Visual Models From Natural Language Supervision (2021, background): “While initial efforts focused primarily on predictive objectives over time research shifted towards learning joint multi-modal embedding spaces with techniques like kernel Canonical Correlation Analysis and various ranking objectives (Weston et al., 2010; Socher & Fei-Fei, 2010; Hodosh et al., 2013)”
Generating Sentences by Editing Prototypes (2017, background): “For example, in the image captioning literature (Hodosh et al., 2013), Kuznetsova (2013) and Mason (2014) proposed to generate image captions by first retrieving a prototype caption based on an image context, and then applying sentence compression to tailor the prototype to a particular image.”
Webly Supervised Joint Embedding for Cross-Modal Image-Text Retrieval (2018, methodology): “to a common space based on extracting a triplet of object, action, and scene is presented. A number of image-text embedding approaches has been developed based on Canonical Correlation Analysis (CCA) [12, 16, 47, 57]. Ranking loss has been used for training the embedding in most recent works relatin”
Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning (2018, background): “Automatic image captioning has a long history (Hodosh et al., 2013; Donahue et al., 2014; Karpathy and Fei-Fei, 2015; Kiros et al., 2015).”
Learning Visual Features from Large Weakly Supervised Data (2015, neutral): “Our work is also related to prior studies on multi-modal embedding [44, 56] that explore approaches such as kernel canonical component analysis [16, 18], restricted Boltzmann machines [45], topic models [22], and log-bilinear models [25].”
Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models (2014, methodology): “While Bleu and Rouge have been used in the past, [3] has argued that such automated evaluation methods are unreliable and do not match human judgements.”
Deep Fragment Embeddings for Bidirectional Image Sentence Mapping (2014, methodology): “We evaluate our image-sentence retrieval performance on Pascal1K [2], Flickr8K [3] and Flickr30K [4] datasets.”
Grounded Compositional Semantics for Finding and Describing Images with Sentences (2014, methodology): “Most related is the very recent work of Hodosh et al. (2013).”
From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions (2014, methodology): “…be of great interest for people who aim to build systems that automatically associate image with sentences that describe them (Farhadi et al., 2010; Kulkarni et al., 2011; Li et al., 2011; Yang et al., 2011; Mitchell et al., 2012; Kuznetsova et al., 2012; Gupta et al., 2012; Hodosh et al., 2013).”
Combined Scaling for Zero-shot Transfer Learning (2021, methodology): “…texts, which is related to many previous works, such as (Hironobu et al., 1999; Weston et al., 2010; Socher and Fei-Fei, 2010; Socher et al., 2013; Hodosh et al., 2013; Frome et al., 2013; Norouzi et al., 2013; Kiros et al., 2014; Socher et al., 2014; Akata et al., 2015b,a; Nam et al., 2017;…”
VQA: Visual Question Answering (2015, background): “For some tasks, such as image captioning, automatic evaluation is still a difficult and open research problem [43, 10, 18].”
Microsoft COCO Captions: Data Collection and Evaluation Server (2015, methodology): “Since automatic evaluation metrics do not always correspond to human judgment, we hope to conduct experiments using human subjects to judge the quality of automatically generated captions, which are most similar to human captions, and whether they are grammatically correct [45], [42], [7], [4], [5].”
Show and tell: A neural image caption generator (2014, background): “We did this in all the experiments (similar to [8]), and it did help quite a lot in terms of generalization.”
Deep Visual-Semantic Alignments for Generating Image Descriptions (2014, neutral): “The standard deviations on the recall values range from approximately 0.5 to 1.0.”
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention (2015, background): “3) on three benchmark datasets: Flickr8k (Hodosh et al., 2013), Flickr30k (Young et al.”