Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics (Extended Abstract)

By Micah Hodosh, Peter Young, Julia Hockenmaier. 2015

52 citations recorded by the source corpus.

Subfields: Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

VQA: Visual Question Answering (2015, background): “For some tasks, such as image captioning, automatic evaluation is still a difficult and open research problem (Vedantam et al. 2015; Elliott and Keller 2014; Hodosh et al. 2013).”
CIDEr: Consensus-based image description evaluation (2014, methodology): “Broadly, the techniques are either retrieval-[11, 27, 15] or generation-based [25, 17, 38, 32].”
Order-Embeddings of Images and Language (2015, methodology): “The caption-image retrieval task has become a standard evaluation of joint models of vision and language (Hodosh et al., 2013; Lin et al., 2014a). The task involves ranking a large dataset of images by relevance for a query caption (Image Retrieval), and ranking captions by relevance for a query ima”
Exploring Nearest Neighbor Approaches for Image Captioning (2015, methodology): “Previously, several papers proposed producing image captions by first finding similar images, and then copying their captions [9], [29], [13].”
SCA-CNN: Spatial and Channel-Wise Attention in Convolutional Networks for Image Captioning (2016, methodology): “For example, Flickr8k’s performance is more easier to degrade than MS COCO when adding more attention layers, as the size of Flickr8k is much smaller than MS COCO.”
Image retrieval using scene graphs (2015, methodology): “We adapt the metrics from [26]; specific lly, f r each method we report th m dian rank of the highest ranked true positive image I2, .”
Commonly Uncommon: Semantic Sparsity in Situation Recognition (2016, background): “For the structured case, image captioning models [45, 22, 7, 11, 33, 20, 35, 31] have been observed to suffer from a lack of diversity and generalization [42].”
Evaluating Object Hallucination in Large Vision-Language Models (2023, background): “…to human instructions, demonstrating strong capacities in solving various vision-language tasks, e.g., image captioning (Ordonez et al., 2011; Hodosh et al., 2015; Sharma et al., 2018; Agrawal et al., 2019) and visual question answering (Antol et al., 2015a; Zhang et al., 2016; Goyal et al.,…”
Re-evaluating Automatic Metrics for Image Captioning (2016, methodology): “These approaches have attained considerable success in the recent benchmarks such as FLICKR 8 K (Hodosh et al., 2013), FLICKR 30 K (Young et al., 2014) and MS COCO (Lin et al., 2014) as compared to the earlier techniques which explicitly detect objects and generate descriptions by using surface…”
Improving Captioning for Low-Resource Languages by Cycle Consistency (2019, background): “Most caption datasets [7–9] in these works are collected in the English language.”
Im2Text and Text2Im: Associating Images and Texts for Cross-Modal Retrieval (2014, methodology): “While [5, 22] do this by matching query image with database images, [12, 26] perform this by matching images and captions in a projected space learned using canonical correlation analysis (CCA).”
Open-book Video Captioning with Retrieve-Copy-Generate Network (2021, methodology): “Reviewing the early retrieval-based visual description methods can be roughly divided into two categories: [21, 9, 16] cast the problem as retrieval from a visual space problem and transfer the retrieved descriptions to a novel image/video; [5, 25, 11] cast the problem as retrieval from a multimodal”
Mind's eye: A recurrent visual representation for image caption generation (2015, methodology): “Various approaches were used to learn the projection, including Kernel Canonical Correlation Analysis (KCCA) [16], recursive neural networks [38], or deep neural networks [20].”