Exploring Human-like Attention Supervision in Visual Question Answering

By Tingting Qiao, Jianfeng Dong, Duanqing Xu. 2017

115 citations recorded by the source corpus.

Attention mechanisms have been widely applied in the Visual Question Answering (VQA) task, as they help to focus on the area-of-interest of both visual and textual information. To answer the questions correctly, the model needs to selectively target different areas of an image, which suggests that an attention-based model may benefit from an explicit attention supervision. In this work, we aim to address the problem of adding attention supervision to VQA models. Since there is a lack of human attention data, we first propose a Human Attention Network (HAN) to generate human-like attention maps, training on a recently released dataset called Human ATtention Dataset (VQA-HAT). Then, we apply the pre-trained HAN on the VQA v2.0 dataset to automatically produce the human-like attention maps for all image-question pairs. The generated human-like attention map dataset for the VQA v2.0 dataset is named as Human-Like ATtention (HLAT) dataset. Finally, we apply human-like attention supervision to an attention-based VQA model. The experiments show that adding human-like supervision yields a more accurate attention together with a better performance, showing a promising future for human-like attention supervision in VQA.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Question Answering & Reading Comprehension

Citations in the corpus

MirrorGAN: Learning Text-To-Image Generation by Redescription (2019, background): “Attention models have been extensively exploited in computer vision and natural language processing, for instance in object detection [21, 6, 18, 41], image/video captioning [34, 9, 31], visual question answering [2, 33, 36, 22], and neural machine translation [19, 8].”
Taking a HINT: Leveraging Explanations to Make Vision and Language Models More Grounded (2019, methodology): “While prior work [22, 17] has shown this approach results in more human-like attention maps, our experiments show it to be ineffective at reducing language bias.”
Learning What Makes a Difference from Counterfactual Examples and Gradient Supervision (2020, methodology): “Some methods also supervise the model’s attention [37,51,57] with ground truth human attention maps [18].”
Improving Natural Language Processing Tasks with Human Gaze-Guided Neural Attention (2020, background): “While neural networks are able to learn meaningful attention mechanisms using only supervision received for the target task, the addition of human gaze information has been shown to be beneﬁcial in many cases [32, 58, 80, 84].”
Learning compositionally through attentive guidance (2018, methodology): “This technique has been tried before in the context of VQA [25,26] and Machine Translation [27].”
Interpretable Visual Question Answering by Visual Grounding From Attention Supervision Mining (2018, methodology): “[18] use the attention labels in the VQAHAT dataset to train an attention proposal network that is able to predict image areas relevant to answer a visual question.”
Attention in Natural Language Processing (2019, background): “This holds for several scenarios, such as visual question answering [137], domain classification for natural language understanding [138], and semantic role labeling [97].”
Grounding Answers for Visual Questions Asked by Visually Impaired People (2022, background): “Modern VQA algorithms often rely on attention maps to determine where to look to find the answers to visual questions, with this having been the dominant approach for a number of years [9, 29, 40].”
Counterfactual Samples Synthesizing for Robust Visual Question Answering (2020, background): “To improve visual-explainable ability, early works [32, 26, 43] directly apply human attention as supervision to guide the models’ attention maps.”
Human Gaze Assisted Artificial Intelligence: A Review (2020, background): “…referring) [Bala-jee Vasudevan et al. , 2018], generating descriptive captions for images [Sugano and Bulling, 2016; Tavakoli et al. , 2017; Cornia et al. , 2018a; Chen and Zhao, 2018; He et al. , 2019b] and videos [Yu et al. , 2017], as well as visual question answering [Qiao et al. , 2018].”