Re-evaluating Automatic Metrics for Image Captioning

By Mert Kilickaya, Aykut Erdem, Nazli Ikizler-Cinbis, Erkut Erdem. 2016

194 citations recorded by the source corpus.

The task of generating natural language descriptions from images has received a lot of attention in recent years. Consequently, it is becoming increasingly important to evaluate such image captioning approaches in an automatic manner. In this paper, we provide an in-depth evaluation of the existing image captioning metrics through a series of carefully designed experiments. Moreover, we explore the utilization of the recently proposed Word Mover’s Distance (WMD) document metric for the purpose of image captioning. Our findings outline the differences and/or similarities between metrics and their relative robustness by means of extensive correlation, accuracy and distraction based evaluations. Our results also demonstrate that WMD provides strong advantages over other metrics.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts (2021, background): “For in-depth discussions of these metrics see [18].”
Why We Need New Evaluation Metrics for NLG (2017, methodology): “Following previous work (Vedantam et al., 2015; Kilickaya et al., 2017), we mainly concentrate on WBMs. Results summarised in Table 4 show that most metrics’ performance is not significantly different from that of a random score (Wilcoxon signed rank test).”
Handling Divergent Reference Texts when Evaluating Table-to-Text Generation (2019, background): “Over the years several studies have evaluated automatic metrics for measuring text generation performance (Callison-Burch et al., 2006; Stent et al., 2005; Belz and Reiter, 2006; Reiter, 2018; Liu et al., 2016; Kilickaya et al., 2017; Gatt and Krahmer, 2018).”
A Structured Review of the Validity of BLEU (2018, background): “For example, Kilickaya et al. (2017) presents BLEU-human correlations on several data sets.”
Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation (2017, background): “Reiter & Belz, 2009; Konstas & Lapata, 2013) and image captioning (see Bernardi et al., 2016; Kilickaya et al., 2017).”
Evaluation of Text Generation: A Survey (2020, methodology): “There are only a few published papers that review evaluation methods for specific NLG tasks, such as image captioning (Kilickaya et al., 2017), machine translation (Goutte, 2006), online review generation (Garbacea et al., 2019), interactive systems (Hastie & Belz, 2014a), and conversational dialog…”
Know More Say Less: Image Captioning Based on Scene Graphs (2019, methodology): “Forevaluation, we use the most commonly used metrics: BLEU [74], METEOR [75], ROUGE-L [76], CIDEr [77], SPICE [78] and WMD [79].”
WMDO: Fluency-based Word Mover’s Distance for Machine Translation Evaluation (2019, methodology): “WMD has been recently used for the evaluation of image captioning models (Kilickaya et al., 2017; Madhyastha et al., 2019).”
Findings of the Second Shared Task on Multimodal Machine Translation and Multilingual Image Description (2017, background): “We also hope to conduct human evaluation to better assess performance because automatic metrics are problematic for this task (Elliott and Keller, 2014; Kilickaya et al., 2017).”
EEGtoText: Learning to Write Medical Reports from EEG Recordings (2019, background): “Since METEOR is always better than BLEU and ROUGE in terms of consistency with human judgement (Kilickaya et al., 2016), here we consider METEOR as our primary metric for evaluating EEG text generation.”
Counterfactual Story Reasoning and Generation (2019, background): “Similar issues arise in the area of controllable language generation (e.g., Hu et al., 2017), which involves preserving the content of text while changing it along a single or multiple dimensions, such as theme (Koncel-Kedziorski et al., 2016), style (Lample et al., 2019), and sentiment (Shen et…”
Improving Automatic Source Code Summarization via Deep Reinforcement Learning (2018, background): “It has always been a tough challenge to evaluate the similarity between two sentences for the tasks such as neural machine translation [37], image captioning [39].”
From Show to Tell: A Survey on Deep Learning-Based Image Captioning (2021, methodology): “For example, the WMD score [161], originally introduced to evaluate document semantic dissimilarity, can also be applied to cap-tioning evaluation by considering generated captions and ground-truth captions as the compared documents [162].”
A Survey of Evaluation Metrics Used for NLG Systems (2020, methodology): “in the reference should sum up to its corresponding value in →−r . Although initially proposed for document classification, WMD has been favourably adopted for evaluating the task of image captioning [70]. WMD has also been adopted for summarization and MT evaluation. However, since WMD is insensiti”
From Show to Tell: A Survey on Image Captioning (2021, methodology): “WMD was introduced to evaluate document semantic dissimilarity but can also be applied to captioning evaluation (converted into a similarity score via a negative exponential) by considering generated captions and ground-truth captions as the compared documents [142].”
Cooperative Generator-Discriminator Networks for Abstractive Summarization with Narrative Flow (2019, background): “Recent work in abstractive summarization has re-ported issues with reference-based automatic metrics such as ROUGE (Hoang et al., 2019; Sun et al., 2019; Clark et al., 2019; Kilickaya et al., 2017).”