SPICE: Semantic Propositional Image Caption Evaluation
By Peter Anderson, Basura Fernando, Mark Johnson, Stephen Gould. 2016
2,344 citations recorded by the source corpus.
There is considerable interest in the task of automatically generating image captions. However, evaluation is challenging. Existing automatic evaluation metrics are primarily sensitive to n-gram overlap, which is neither necessary nor sufficient for the task of simulating human judgment. We hypothesize that semantic propositional content is an important component of human caption evaluation, and propose a new automated caption evaluation metric defined over scene graphs coined SPICE. Extensive evaluations across a range of models and datasets indicate that SPICE captures human judgments over model-generated captions better than other automatic metrics (e.g., system-level correlation of 0.88 with human judgments on the MS COCO dataset, versus 0.43 for CIDEr and 0.53 for METEOR). Furthermore, SPICE can answer questions such as which caption-generator best understands colors? and can caption-generators count?
Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Ming-Hsuan Yang
- Jitendra Malik
- Cordelia Schmid
- Yi Yang
In Multimodal Learning
In Evaluation & Benchmarks
- Mohit Bansal
- Yejin Choi
- Jianfeng Gao
- Li Fei-Fei
- Trevor Darrell
- Elizabeth Clark
- Jitendra Malik
- Abhinav Kumar Gupta
Citations in the corpus
- TAP: Text-Aware Pre-training for Text-VQA and Text-Caption (2020, methodology): “The cap-tions are evaluated by the caption metrics (BLEU [42], ME-TEOR [14], ROUGE L [35], SPICE [3], and CIDEr [52]).”
- ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph (2020, methodology): “Given the text sentence w , we parse it into a scene graph [29], which denotes as G ( w ) = < O ( w ) , E ( w ) , K ( w ) > , where O ( w ) is the set of object mentioned in w , E ( w ) ⊆ O ( w ) × R ( w ) × O ( w ) is the set of hyper-edges representing relationships nodes between object nodes and…”
- VirTex: Learning Visual Representations from Textual Annotations (2020, methodology): “We report image captioning performance (CIDEr [105] and SPICE [106]) of VirTex models on COCO val2017.”
- Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning (2018, methodology): “7) test sets, we report numerical values for the CIDEr, ROUGE-L, and SPICE (Anderson et al., 2016) § .”
- Object-Centric Image Generation from Layouts (2020, methodology): “Scene-graphs are often used as intermediate representations in image captioning (Yang et al. 2019; Anderson et al. 2016), reconstruction (Gu et al.”
- How Much Can CLIP Benefit Vision-and-Language Tasks? (2021, methodology): “0, 6.5 CIDEr on COCO Captioning, and 4.0% success rate on Room-to-Room navigation.”
- On the Evaluation of Vision-and-Language Navigation Instructions (2021, methodology): “For system-level evaluations with reference instructions, we recommend SPICE (Anderson et al., 2016).”
- Is Reinforcement Learning (Not) for Natural Language Processing?: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization (2022, methodology): “, 2015), SPICE (Anderson et al., 2016) and SPiDer (Liu et al.”
- UFO: A UniFied TransfOrmer for Vision-Language Representation Learning (2021, methodology): “The beam search size is set as 1, and the accuracy is evaluated with BLEU@4 [34], METEOR [9], CIDEr [46], and SPICE [3].”
- ClipCap: CLIP Prefix for Image Captioning (2021, positive): “Similar to Li et al. [19], we validate our results over the COCO dataset using the common metrics BLEU [27], METEOR [10], CIDEr [37] and SPICE [3], and for the nocaps dataset using CIDEr and SPICE.”
- Understanding and Evaluating Racial Biases in Image Captioning (2021, methodology): “Table 1: The differences in captioning performance (score on lighter - score on darker) as measured by BLEU [55], CIDEr [68], and SPICE [2] multiplied by 100 on the COCO 2014 validation dataset.”
- VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning (2021, methodology): “The evaluation metrics include BLEU [61], METEOR [5], ROUGE[49], CIDEr [80] and SPICE [2].”
- Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts (2021, methodology): “Evaluation metrics To measure the performance on image caption generation, we consider the standard metrics BLEU-1,4 [59], ROUGE-L [48], METEOR [10], CIDEr-D [74], and SPICE [4].”
- Unifying Vision-and-Language Tasks via Text Generation (2021, methodology): “, 2015), METEOR (Banerjee & Lavie, 2005), SPICE (Anderson et al., 2016) as evaluation metrics using COCOEvalCap implementation.”
- Attention on Attention for Image Captioning (2019, methodology): “We use different metrics, including BLEU [29], METEOR [33], ROUGE-L [13], CIDErD [36] and SPICE [1], to evaluate the proposed method and compare with other methods.”
- Language Is Not All You Need: Aligning Perception with Language Models (2023, methodology): “We use COCOEvalCap4 to compute CIDEr [VLZP15] and SPICE [AFJG16] scores as the evaluation metrics.”
- Scaling Autoregressive Models for Content-Rich Text-to-Image Generation (2022, methodology): “Following DALL-Eval [55], we also measure text-image fit through automated captioning evaluation (or captioner evaluation): an image output by the model is captioned with a pretrained VL-T5 model [56] and then the similarity of the input prompt and and the generated caption is assessed via BLEU [57]”
- GIT: A Generative Image-to-text Transformer for Vision and Language (2022, methodology): “Our model achieves new SOTA performance 1) on all metrics (BLEU@4 [73], METEOR [13], CIDEr [90], SPICE [3]) with SCST and 2) on CIDEr and BLEU@4 with cross-entropy optimization only.”
- Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering (2017, methodology): “To evaluate caption quality, we use the standard automatic evaluation metrics, namely SPICE [1], CIDEr [42], METEOR [8], ROUGE-L [22] and BLEU [29].”