Attention Correctness in Neural Image Captioning

By Chenxi Liu, Junhua Mao, Fei Sha, Alan L. Yuille. 2016

229 citations recorded by the source corpus.

Attention mechanisms have recently been introduced in deep learning for various tasks in natural language processing and computer vision. But despite their popularity, the ``correctness'' of the implicitly-learned attention maps has only been assessed qualitatively by visualization of several examples. In this paper we focus on evaluating and improving the correctness of attention in neural image captioning models. Specifically, we propose a quantitative evaluation metric for the consistency between the generated attention maps and human annotations, using recently released datasets with alignment between regions in images and entities in captions. We then propose novel models with different levels of explicit supervision for learning attention maps during training. The supervision can be strong when alignment between regions and caption entities are available, or weak when only object segments and categories are provided. We show on the popular Flickr30k and COCO datasets that introducing supervision of attention maps during training solidly improves both attention correctness and caption quality, showing the promise of making machine perception more human-like.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Semantic Compositional Networks for Visual Captioning (2016, methodology): “This work was followed by [53] which introduced a review module to improve the attention mechanism and [28] which proposed a method to improve the correctness of visual attention.”
Grounded Video Description (2018, background): “While researchers have started to discover and study these problems for image description [14, 9, 24, 20],1 they are even more pronounced”
Taking a HINT: Leveraging Explanations to Make Vision and Language Models More Grounded (2019, neutral): “While prior work [22, 17] has shown this approach results in more human-like attention maps, our experiments show it to be ineffective at reducing language bias.”
A Comprehensive Survey of Deep Learning for Image Captioning (2018, methodology): “Liu et al. [84] proposed a method for neural image captioning.”
Learning What Makes a Difference from Counterfactual Examples and Gradient Supervision (2020, methodology): “All of these methods are speciﬁc to VQA or to captioning [30,37] whereas we describe a much more general approach.”
Counterfactual Samples Synthesizing for Robust Visual Question Answering (2020, background): “To improve visual-explainable ability, early works [32, 26, 43] directly apply human attention as supervision to guide the models’ attention maps.”
MDNet: A Semantically and Visually Interpretable Medical Image Diagnosis Network (2017, background): “As indicted by [20], improving such alignment can be achieved by adding supervision on attention maps by using regionlevel labels (e.”
On the Potential of Lexico-logical Alignments for Semantic Parsing to SQL Queries (2020, unknown): “Attention and Alignments Explicit supervision for attention mechanisms (Bahdanau et al., 2015) is helpful for many tasks, including machine translation (Liu et al., 2016; Mi et al., 2016), image captioning (Liu et al., 2017), and visual question answering (Gan et al., 2017).”
Understanding Attention and Generalization in Graph Neural Networks (2019, methodology): “We evaluate attention correctness using area under the ROC curve (AUC) as an alternative to other methods, such as [27], which can be overoptimistic in some extreme cases, such as when all attention is concentrated in a single node or attention is uniformly spread over all nodes.”
Top-Down Visual Saliency Guided by Captions (2016, methodology): “In Table 2 we also provide a direct comparison of our attention method to the soft attention model [27] in terms of the attention correctness metric proposed in [14].”
Stack-Captioning: Coarse-to-Fine Learning for Image Captioning (2017, methodology): “2016a) and (Liu et al. 2017b) which further improves the visual attention mechanism.”
Scene Graph Parsing as Dependency Parsing (2018, background): “…image retrieval (Johnson et al., 2015; Vendrov et al., 2015), image captioning (Mao et al., 2014; Karpathy and Li, 2015; Donahue et al., 2015; Liu et al., 2017b), visual question answering (Antol et al., 2015; Zhu et al., 2016; Andreas et al., 2016), and referring expressions (Hu et al.,…”
Areas of Attention for Image Captioning (2016, methodology): “ to be stable across meaningful sub-sequences, such as noun phrases ( e.g .A couple of elephants ) and verb phrases ( e.g .is ying. ). Attention correctness. We follow the approach of Liu et al . [21] to quantitatively assess the alignment of attentionwithimageregionscorrespondingtothegeneratedcapti”
A General Survey on Attention Mechanisms in Deep Learning (2022, methodology): “In [117], the concept of attention correctness is proposed, which is a quantitative intrinsic performance metric that evaluates the quality of the attention mechanism based on”
Mitigating Gender Bias in Captioning Systems (2020, methodology): “We adopt two evaluation metrics to calculate the similarity: Pointing Game [43], which measures whether the point with highest attention value falls in the person segmentation masks, and Attention Sum [22], which calculates the sum of attention weights that fall in the person regions.”