Focused Evaluation for Image Description with Binary Forced-Choice Tasks

By Micah Hodosh, Julia Hockenmaier. 2016

34 citations recorded by the source corpus.

Current evaluation metrics for image description may be too coarse. We therefore propose a series of binary forced-choice tasks that each focus on a different aspect of the captions. We evaluate a number of different off-the-shelf image description systems. Our results indicate strengths and shortcomings of both generation and ranking based approaches.

Subfields: Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

ShapeWorld - A new test methodology for multimodal language understanding (2017, background): “, 2015) or systematic modification of captions (Hodosh and Hockenmaier, 2016).”
Why We Need New Evaluation Metrics for NLG (2017, background): “This way, we can account for equally valid system outputs, which is one of the main drawbacks of forced-choice approaches (Hodosh and Hockenmaier, 2016).”
A Survey of Evaluation Metrics Used for NLG Systems (2020, background): “That is, while the permuted sentence need not necessarily make sense, BLUE assigns the same score to it as long as the number of matching n-grams doesn’t change. More recently, Hodosh and Hockenmaier [62] provide a framework to create incorrect captions by replacing people (or the scene) in the capt”
From Show to Tell: A Survey on Deep Learning-Based Image Captioning (2021, background): “In this sense, the design of appropriate and reproducible evaluation protocols [251], [252], [253] and insightful metrics remains an open challenge in image captioning.”
Fine-Grained Video-Text Retrieval With Hierarchical Graph Reasoning (2020, methodology): “To further evaluate our model on ﬁne-grained retrieval, we propose a new binary selection task [14, 15] which requires the system to select the correct matching sentence for the given video from two similar candidate sentences with subtle difference.”
Re-evaluating Automatic Metrics for Image Captioning (2016, methodology): “For this purpose, we employ the binary (two-alternative) forced choice task introduced in (Hodosh and Hockenmaier, 2016) to compare the existing image captioning models.”