Comparing Automatic Evaluation Measures for Image Description

By Desmond Elliott, Frank Keller. 2014

136 citations recorded by the source corpus.

Image description is a new natural language generation task, where the aim is to generate a human-like description of an image. The evaluation of computer-generated text is a notoriously difficult problem, however, the quality of image descriptions has typically been measured using unigram BLEU and human judgements. The focus of this paper is to determine the correlation of automatic measures with human judgements for this task. We estimate the correlation of unigram and Smoothed BLEU, TER, ROUGE-SU4, and Meteor against human judgements on two data sets. The main finding is that unigram BLEU has a weak correlation, and Meteor has the strongest correlation with human judgements.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Evaluation & Benchmarks

Citations in the corpus

VQA: Visual Question Answering (2015, background): “For some tasks, such as image captioning, automatic evaluation is still a difficult and open research problem [43, 10, 18].”
Microsoft COCO Captions: Data Collection and Evaluation Server (2015, background): “Designing effective automatic evaluation metrics that are highly correlated with human judgment remains a difﬁcult challenge [7], [42], [45], [46].”
CIDEr: Consensus-based image description evaluation (2014, methodology): “Recently, METEOR has been used for image description evaluation with more promising results [12].”
SPICE: Semantic Propositional Image Caption Evaluation (2016, methodology): “te and inexpensive evaluation metrics to encourage rapid progress. Unfortunately, existing metrics have proven to be inadequate substitutes for human judgment in the task of evaluating image captions [7,3,8]. As such, there is an urgent need to develop new automated evaluation metrics for this task ”
From captions to visual concepts and back (2014, background): “All of these automatic metrics are known to only roughly correlate with human judgment [10].”
Translating Videos to Natural Language Using Deep Recurrent Neural Networks (2014, background): “Propelled by successes of deep learning, several groups released record breaking re-sults in just the past year (Donahue et al., 2014; Mao et al., 2014; Karpathy et al., 2014; Fang et al., 2014; Kiros et al., 2014; Vinyals et al., 2014; Kuznetsova et al., 2014).”
Learning a Recurrent Visual Representation for Image Caption Generation (2014, background): “…gives BLEU-1 to BLEU-4 (with penalty) as 60.4%, 26.4%, 12.6% and 6.5%, compared to human consistency 65.9%, 30.5%, 13.6% and It is known that automatic measures are only roughly correlated with human judgment [5], so it is also important to evaluate the generated sentences using human studies.”
Exploring Nearest Neighbor Approaches for Image Captioning (2015, background): “However, given the strong BLEU/CIDEr performance of the nearest neighbor systems, this provides additional evidence that automatic metrics may only be a rough estimate of human judgments, as also noted in [7], [13], [34].”
TreeTalk: Composition and Compression of Trees for Image Descriptions (2014, background): “While 4-gram BLEU with brevity penalty is found to correlate better with human judges by recent studies (Elliott and Keller, 2014), we found that this is not the case for our task.”
Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation (2017, background): “As for the relationship between metrics and human judgements, Elliott and Keller (2013) concluded that meteor correlates better than bleu (see Elliott & Keller, 2014, for a systematic comparison of automatic metrics in this domain), a finding also confirmed in their later work (Elliott & De Vries,…”
A Shared Task on Multimodal Machine Translation and Crosslingual Image Description (2016, background): “Meteor is the primary evaluation measure because it has been shown to have a much stronger correlation with human judgements than BLEU or TER for this task (Elliott and Keller, 2014).”
Don't just listen, use your imagination: Leveraging visual common sense for non-visual tasks (2015, background): “However, these tasks are difficult to evaluate [29, 13] or often evaluate aspects of the problem that are less relevant to image understanding e.”
Reinforced Video Captioning with Entailment Rewards (2017, background): “Several papers have presented the relative comparison of image captioning metrics, and their pros and cons (Vedantam et al., 2015; Anderson et al., 2016; Liu et al., 2016b; Hodosh et al., 2013; Elliott and Keller, 2014).”
A Survey of Evaluation Metrics Used for NLG Systems (2020, methodology): “various NLG tasks as summarized in Table4. Work Metrics Task Datasets on which poor correlation was observed [114] BLEU, TER, ROUGE, NIST, LEPOR, METEOR, CIDEr DG BAGEL [103], SF-HOTEL, SF-REST [167] [42] Smoothed-BLEU [25], TER, ROUGE-SU4, Meteor IC Flickr8K[63], E& K[41] [72] ROUGE-1,2,L AS CN”
Mind's eye: A recurrent visual representation for image caption generation (2015, background): “It is known that automatic measures are only roughly correlated with human judgment [8, 40, 16], so it is also important to evaluate the generated sentences using human studies.”
FOIL it! Find One mismatch between Image and Language caption (2017, methodology): “Thirdly, existing IC evaluation metrics are sensitive to n-gram overlap and there is a need for measures that better simulate human judgments (Hodosh et al., 2013; Elliott and Keller, 2014; Anderson et al., 2016).”