Differential Attention for Visual Question Answering

By Badri N. Patro, Vinay P. Namboodiri. 2018

79 citations recorded by the source corpus.

In this paper we aim to answer questions based on images when provided with a dataset of question-answer pairs for a number of images during training. A number of methods have focused on solving this problem by using image based attention. This is done by focusing on a specific part of the image while answering the question. Humans also do so when solving this problem. However, the regions that the previous systems focus on are not correlated with the regions that humans focus on. The accuracy is limited due to this drawback. In this paper, we propose to solve this problem by using an exemplar based method. We obtain one or more supporting and opposing exemplars to obtain a differential attention region. This differential attention is closer to human attention than other image based attention methods. It also helps in obtaining improved accuracy when answering questions. The method is evaluated on challenging benchmark datasets. We perform better than other image based attention methods and are competitive with other state of the art methods that focus on both image and questions.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Zhe Gan
2. Linjie Li
3. Jingjing Liu
4. Yu Cheng
5. Qi Wu
6. Jianfeng Gao
7. Mohit Bansal
8. Badri N. Patro

In Computer Vision and Pattern Recognition

1. Zhe Gan
2. Luc Van Gool
3. Dacheng Tao
4. Ming-Hsuan Yang
5. Linjie Li
6. Cordelia Schmid
7. Jingjing Liu
8. Yu Cheng

Citations in the corpus

• Global-and-local attention networks for visual recognition (2018, background): “Such mechanisms were the focus of early systems for image categorization [16], and were later shown to yield significant improvements for visual question answering (VQA) and captioning [17–22].”
• Semantic Equivalent Adversarial Data Augmentation for Visual Question Answering (2020, background): “There are fewer works [18,33,35,1,30] that learn data augmentation for VQA.”
• U-CAM: Visual Explanation Using Uncertainty Based Class Activation Maps (2019, methodology): “[43, 54, 33, 38] have proposed attention-based methods for the explanation in VQA, which use question to attend over specific regions in an image.”
• Attending to Discriminative Certainty for Domain Adaptation (2019, methodology): “Attention-based networks have been widely applied in many computer vision applications such as image captioning [51, 53], visual question answer [52, 31, 18] and speech recognition [6].”
• Dual Visual Attention Network for Visual Dialog (2019, background): “…image captioning [Karpathy and Fei-Fei, 2015; Xu et al. , 2015; Lu et al. , 2017b], referring expressions [Hu et al. , 2016; Zhang et al. , 2018], visual question answering (VQA) [An-tol et al. , 2015; Patro and Namboodiri, 2018; Anderson et al. , 2018], and visual dialog [Das et al. , 2017].”
• Relation-Aware Graph Attention Network for Visual Question Answering (2019, background): “In lieu of directly using visual features from CNN-based feature extractors, [56, 11, 41, 33, 49, 38, 63, 36] explored various image attention mechanisms to locate regions that are relevant to the question.”