Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering

By Huijuan Xu, Kate Saenko. 2015

787 citations recorded by the source corpus.

We address the problem of Visual Question Answering (VQA), which requires joint image and language understanding to answer a question about a given photograph. Recent approaches have applied deep image captioning methods based on convolutional-recurrent networks to this problem, but have failed to model spatial inference. To remedy this, we propose a model we call the Spatial Memory Network and apply it to the VQA task. Memory networks are recurrent neural networks with an explicit attention mechanism that selects certain parts of the information stored in memory. Our Spatial Memory Network stores neuron activations from different spatial regions of the image in its memory, and uses the question to choose relevant regions for computing the answer, a process of which constitutes a single "hop" in the network. We propose a novel spatial attention architecture that aligns words with image patches in the first hop, and obtain improved results by adding a second attention hop which considers the whole question to choose visual evidence based on the results of the first hop. To better understand the inference process learned by the network, we design synthetic questions that specifically require spatial inference and visualize the attention weights. We evaluate our model on two published visual question answering datasets, DAQUAR [1] and VQA [2], and obtain improved results compared to a strong deep baseline model (iBOWIMG) which concatenates image and question features to predict the answer [3].

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Bidirectional Attention Flow for Machine Comprehension (2016, methodology): “At a ﬁner level, each question word attends to each image patch and the highest attention value for each spatial location (Xu & Saenko, 2016) is adopted.”
Measuring and Improving BERT’s Mathematical Abilities by Predicting the Order of Reasoning. (2021, background): “BERT’s strong performance has resulted in the Cambrian explosion of studies of the inner working mechanisms and various modiﬁca-tions (Clark et al., 2019; de Vries et al., 2019; Lan et al., 2019; Liu et al., 2019; Sanh et al., 2019; Rad-ford et al.; Raffel et al., 2019; Yang et al., 2019).”
Towards VQA Models That Can Read (2019, background): “VQA models typically use some variant of attention to get a representation of the image that is relevant for answering the given question [2, 7, 30, 47, 48, 51, 17].”
A simple neural network module for relational reasoning (2017, positive): “eural networks. Next, both representations are combined, and the answers are either predicted or generated. Most successful methods also use an attention mechanism that locate important image regions [5, 44, 45, 46]. In our work, we follow a similar pipeline, but we use Relation Networks as a powerf”
Just Ask: Learning to Answer Questions from Millions of Narrated Videos (2020, neutral): “Various approaches to combine spatial image representations and sequential question representations have been proposed [7, 10, 25, 57, 86, 88, 91].”
OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge (2019, background): “Attention-based models [12, 31, 46, 47, 48, 54] better guide the model in answering the questions by highlighting image regions that are relevant to the question.”
Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering (2016, background): “have proposed visual question answering datasets [6,31,35, 41,14,57,48,46] and models [13,34,5,54,33,36,59,56,55, 52,45,27,38,20,53,43,23].”
Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding (2016, methodology): “Stacked Attention Networks (Yang et al., 2015) and Spatial Memory Networks (Xu et al., 2015) use LSTMs or extract soft-attention on the image features, but ultimately use element-wise product or element-wise sums to merge modalities.”
Rationalizing Neural Predictions (2016, background): “Attention based models offer another lens to the inner workings of neural models (Bahdanau et al., 2015; Cheng et al., 2016; Martins and Astudillo, 2016; Chen et al., 2015; Xu and Saenko, 2015; Yang et al., 2015).”
Bilinear Attention Networks (2018, background): “Xu and Saenko [36] proposed the spatial memory network model estimating the correlation among every image patches and tokens in a sentence.”
Hierarchical Question-Image Co-Attention for Visual Question Answering (2016, methodology): “For open-ended test scenario, we compare our method with the recent proposed SMem [22], SAN [23], FDA [10] and DMN+ [21].”
Modulating early visual processing by language (2017, background): “Another line of research has investigated the role of attention mechanisms in VQA [26, 11, 28].”
Learning to Compose Neural Networks for Question Answering (2016, methodology): “, 2015), or use the question to compute an attention over the input image, and then answer based on both the question and the image features attended to (Yang et al., 2015; Xu and Saenko, 2015).”
Dynamic Memory Networks for Visual and Textual Question Answering (2016, methodology): “Attention mechanisms have been successful components in both machine translation (Bahdanau et al., 2015; Luong module and memory module, to improve question answering.”
Simple Baseline for Visual Question Answering (2015, background): “Recently, several papers have appeared on arXiv (after CVPR’16 submission deadline) proposing neural network architectures for visual question answering, such as [13, 17, 5, 18, 16, 3, 11, 1].”