Paying More Attention to Saliency: Image Captioning with Saliency and Context Attention

By Marcella Cornia, Lorenzo Baraldi, Giuseppe Serra, Rita Cucchiara. 2017

100 citations recorded by the source corpus.

Image captioning has been recently gaining a lot of attention thanks to the impressive achievements shown by deep captioning architectures, which combine Convolutional Neural Networks to extract image representations, and Recurrent Neural Networks to generate the corresponding captions. At the same time, a significant research effort has been dedicated to the development of saliency prediction models, which can predict human eye fixations. Despite saliency information could be useful to condition an image captioning architecture, by providing an indication of what is salient and what is not, no model has yet succeeded in effectively incorporating these two techniques. In this work, we propose an image captioning approach in which a generative recurrent neural network can focus on different parts of the input image during the generation of the caption, by exploiting the conditioning given by a saliency prediction model on which parts of the image are salient and which are contextual. We demonstrate, through extensive quantitative and qualitative experiments on large scale datasets, that our model achieves superior performances with respect to different image captioning baselines with and without saliency.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

SMArT: Training Shallow Memory-aware Transformers for Robotic Explainability (2019, methodology): “anisms [21], the visual representation has turned into a time-varying vector extracted from a grid of CNN features using the hidden state of the language model as the query [22], [23], [6], [24], [8].”
Human Gaze Assisted Artificial Intelligence: A Review (2020, background): “…(object referring) [Bala-jee Vasudevan et al. , 2018], generating descriptive captions for images [Sugano and Bulling, 2016; Tavakoli et al. , 2017; Cornia et al. , 2018a; Chen and Zhao, 2018; He et al. , 2019b] and videos [Yu et al. , 2017], as well as visual question answering [Qiao et al. ,…”
From Show to Tell: A Survey on Deep Learning-Based Image Captioning (2021, unknown): “Subsequent works on this line [54], [55], [56], [57] employed saliency maps as a form of additional attention source.”
Show, Control and Tell: A Framework for Generating Controllable and Grounded Captions (2018, background): “Generally, all integrate recurrent neural networks as language models, and a representation of the image which might be given by the output of one or more layer of a CNN [43, 10, 37, 24], or by a time-varying vector extracted with an attention mechanism [46, 48, 24, 7, 3] selected either from a grid”
From Show to Tell: A Survey on Image Captioning (2021, background): “Subsequent works on this line [42], [43], [44] used predicted saliency information in place of eye-fixation information.”