SPICE: Semantic Propositional Image Caption Evaluation

By Peter Anderson, Basura Fernando, Mark Johnson, Stephen Gould. 2016

2,344 citations recorded by the source corpus.

There is considerable interest in the task of automatically generating image captions. However, evaluation is challenging. Existing automatic evaluation metrics are primarily sensitive to n-gram overlap, which is neither necessary nor sufficient for the task of simulating human judgment. We hypothesize that semantic propositional content is an important component of human caption evaluation, and propose a new automated caption evaluation metric defined over scene graphs coined SPICE. Extensive evaluations across a range of models and datasets indicate that SPICE captures human judgments over model-generated captions better than other automatic metrics (e.g., system-level correlation of 0.88 with human judgments on the MS COCO dataset, versus 0.43 for CIDEr and 0.53 for METEOR). Furthermore, SPICE can answer questions such as which caption-generator best understands colors? and can caption-generators count?

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Trevor Darrell
  4. Xiaogang Wang
  5. Ming-Hsuan Yang
  6. Jitendra Malik
  7. Cordelia Schmid
  8. Yi Yang

In Multimodal Learning

  1. Zhe Gan
  2. Mohit Bansal
  3. Lijuan Wang
  4. Jianfeng Gao
  5. Devi Parikh
  6. Linjie Li
  7. Zicheng Liu
  8. Jianfeng Wang

In Evaluation & Benchmarks

  1. Mohit Bansal
  2. Yejin Choi
  3. Jianfeng Gao
  4. Li Fei-Fei
  5. Trevor Darrell
  6. Elizabeth Clark
  7. Jitendra Malik
  8. Abhinav Kumar Gupta

Citations in the corpus