Egoshots, an ego-vision life-logging dataset and semantic fidelity metric to evaluate diversity in image captioning models
By Pranav Agarwal, Alejandro Betancourt, V. Panagiotou, Vana Panagiotou, Natalia Díaz Rodríguez. 2020
12 citations recorded by the source corpus.
Image captioning models have been able to generate grammatically correct and human understandable sentences. However most of the captions convey limited information as the model used is trained on datasets that do not caption all possible objects existing in everyday life. Due to this lack of prior information most of the captions are biased to only a few objects present in the scene, hence limiting their usage in daily life. In this paper, we attempt to show the biased nature of the currently existing image captioning models and present a new image captioning dataset, Egoshots, consisting of 978 real life images with no captions. We further exploit the state of the art pre-trained image captioning and object recognition networks to annotate our images and show the limitations of existing works. Furthermore, in order to evaluate the quality of the generated captions, we propose a new image captioning metric, object based Semantic Fidelity (SF). Existing image captioning metrics can evaluate a caption only in the presence of their corresponding annotations; however, SF allows evaluating captions generated for images without annotations, making it highly useful for real life generated captions.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Andy Zeng
- Michael S. Ryoo
- Vikas Sindhwani
- Krzysztof Marcin Choromanski
- Peter R. Florence
- Tobias Hinz
- Stefan Wermter
- Adrian S. Wong
In Computer Vision and Pattern Recognition
- Dacheng Tao
- Andy Zeng
- Luc Van Gool
- Kristen Grauman
- Li Fei-Fei
- Jitendra Malik
- Andrew Zisserman
- Trevor Darrell
In Multimodal Learning
- Stefan Wermter
- Michael S. Ryoo
- Stefan Heinrich
- Tobias Hinz
- Andy Zeng
- Jaemin Cho
- Mohit Bansal
- Abhaysinh Zala
Citations in the corpus
- Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022, positive): “Image captioning models are biased based on the datasets they are trained on, and have shown to perform poorly on egocentric images (Agarwal et al. 2020), which aligns with our observations.”
- Semantic Object Accuracy for Generative Text-to-Image Synthesis (2019, background): “Some of these limitations might potentially be overcome in the future by novel image caption evaluation metrics that focus more on objects and semantic content in the scene [54], [56], [59].”