CIDEr: Consensus-based image description evaluation

By Ramakrishna Vedantam, C. Lawrence Zitnick, Devi Parikh. 2014

5,568 citations recorded by the source corpus.

Automatically describing an image with a sentence is a long-standing challenge in computer vision and natural language processing. Due to recent progress in object detection, attribute classification, action recognition, etc., there is renewed interest in this area. However, evaluating the quality of descriptions has proven to be challenging. We propose a novel paradigm for evaluating image descriptions that uses human consensus. This paradigm consists of three main parts: a new triplet-based method of collecting human annotations to measure consensus, a new automated metric that captures consensus, and two new datasets: PASCAL-50S and ABSTRACT-50S that contain 50 sentences describing each image. Our simple metric captures human judgment of consensus better than existing metrics across sentences generated by various sources. We also evaluate five state-of-the-art image description approaches using this new protocol and provide a benchmark for future comparisons. A version of CIDEr named CIDEr-D is available as a part of MS COCO evaluation server to enable systematic evaluation and benchmarking.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

NewsQA: A Machine Comprehension Dataset (2016, methodology): “ase matches (n-grams) against the reference sentence (Papineni et al., 2002). CIDEr was designed to correlate better with human judgements of sentence similarity, and uses tf-idf scores over n-grams (Vedantam et al., 2015). As given in Table 4, humans averaged 0.694 F1 on NewsQA. The human EM scores”
Semantic Noise Matters for Neural Natural Language Generation (2019, methodology): “We use freely available word-overlap-based evaluation metrics (WOM) scripts that come with the E2E data (Duˇsek et al., 2019), 9 supporting BLEU (Papineni et al., 2002), NIST (Doddington, 2002), ROUGE-L (Lin, 2004), METEOR (Lavie and Agarwal, 2007) and CIDEr (Vedantam et al., 2015).”
Neural Generation for Czech: Data and Baselines (2019, methodology): “See Section 4.2 for metrics; scores are averaged over 5 different random ini-tializations, all scores except for NIST and CIDEr are percentages.”
The E2E Dataset: New Challenges For End-to-End Generation (2017, background): “BLEU (Papineni et al., 2002) 0.6925 NIST (Doddington, 2002) 8.4781 METEOR (Lavie and Agarwal, 2007) 0.4703 ROUGE-L (Lin, 2004) 0.7257 CIDEr (Vedantam et al., 2015) 2.3987 Table 6: TGen results on the development set. cent E2E data-driven systems.”
Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions (2020, methodology): “arthasarathi and Pineau, 2018). Some commonly used wordoverlap metrics include F1 (Rajpurkar et al., 2016), BLEU (Papineni et al., 2002; Li et al., 2017c), ROUGE (Lin, 2004; Fan et al., 2019), CIDEr (Vedantam, Zitnick, and Parikh, 2015; Zhou et al., 2020), and METEOR (Banerjee and Lavie, 2005; Zhou ”
Prefix-Tuning: Optimizing Continuous Prompts for Generation (2021, methodology): “, 2002), NIST (Belz and Reiter, 2006), METEOR (Lavie and Agarwal, 2007), ROUGE-L (Lin, 2004), and CIDEr (Vedantam et al., 2015).”
VideoBERT: A Joint Model for Video and Language Representation Learning (2019, methodology): “Besides the BLEU and ME-TEOR metrics used by [39], we also report ROUGE-L [13] and CIDEr [28].”
TAP: Text-Aware Pre-training for Text-VQA and Text-Caption (2020, methodology): “Table 3 shows the CIDEr score on the TextCaps dataset [46].”
Contrastive Learning of Medical Visual Representations from Paired Images and Text (2020, methodology): “For the captioning-based methods, we always use the model checkpoints that achieve the best CIDEr score (Vedantam et al., 2015) on a held-out validation set.”
VirTex: Learning Visual Representations from Textual Annotations (2020, methodology): “We report image captioning performance (CIDEr [105] and SPICE [106]) of VirTex models on COCO val2017.”
The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes (2020, background): “They often rely on bounding boxes or similar features for maximum performance, and are not always easy to evaluate [77].”
Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning (2018, methodology): “5 show CIDEr scores in the 1.02-1.04 range, for both RNN-and Transformer-based models; the scores drop in the 0.35-0.41 range (CIDEr) for the Conceptual-based models tested against COCO groundtruth.”
Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations (2016, methodology): “We calculate BLEU, CIDEr, and METEOR scores (Chen et al., 2015) between the generated descriptions and their ground-truth descriptions.”
On the Evaluation of Vision-and-Language Navigation Instructions (2021, methodology): “We discover that BLEU (Papineni et al., 2002), ROUGE (Lin, 2004), METEOR (Denkowski and Lavie, 2014) and CIDEr (Vedantam et al., 2015) are ineffective for evaluating grounded navigation instructions.”
Is Reinforcement Learning (Not) for Natural Language Processing?: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization (2022, methodology): “, 2020) which generally provide higher correlation with human judgment; 3) task-specific metrics such as CIDER (Vedantam et al., 2015), SPICE (Anderson et al.”
On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgetting (2022, background): “, 2019), coming up with complete or perfect rewards is highly nontrivial (Wu et al., 2016; Vedantam et al., 2015).”
Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-control (2016, background): “Designing an appropriate reward deﬁnition is highly non-trivial, and often the hand-crafted rewards cannot be fully trusted (Vedantam et al., 2015; Liu et al., 2016).”