Generating captions without looking beyond objects

By Hendrik Heuer, Christof Monz, Arnold M. W. Smeulders. 2016

18 citations recorded by the source corpus.

This paper explores new evaluation perspectives for image captioning and introduces a noun translation task that achieves comparative image caption generation performance by translating from a set of nouns to captions. This implies that in image captioning, all word categories other than nouns can be evoked by a powerful language model without sacrificing performance on n-gram precision. The paper also investigates lower and upper bounds of how much individual word categories in the captions contribute to the final BLEU score. A large possible improvement exists for nouns, verbs, and prepositions.

Subfields: Computer Vision and Pattern Recognition, Language and Linguistics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Shortcut learning in deep neural networks (2020, background): “DNNs can generate a plausible caption for an image, but—worryingly—they can do so without ever looking at that image [11].”
The “Something Something” Video Database for Learning and Evaluating Visual Common Sense (2017, background): “Recently, [38, 9] showed how captioning models can “cheat” by generating sensible sentences without necessarily understanding an observed scene in detail.”
ON THE EFFECTIVENESS OF TASK GRANULARITY FOR TRANSFER LEARNING (2018, background): “It has been shown, for example, that a language model trained on unordered, individual words (e.g., object-nouns) predicted by a separate NN can compete with captioning model trained on the actual task end-to-end (e.g., Yao et al. (2015); Heuer et al. (2016)).”