Stacked Attention Networks for Image Question Answering

By Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Smola. 2015

2,013 citations recorded by the source corpus.

This paper presents stacked attention networks (SANs) that learn to answer natural language questions from images. SANs use semantic representation of a question as query to search for the regions in an image that are related to the answer. We argue that image question answering (QA) often requires multiple steps of reasoning. Thus, we develop a multiple-layer SAN in which we query an image multiple times to infer the answer progressively. Experiments conducted on four image QA data sets demonstrate that the proposed SANs significantly outperform previous state-of-the-art approaches. The visualization of the attention layers illustrates the progress that the SAN locates the relevant visual clues that lead to the answer of the question layer-by-layer.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Bidirectional Attention Flow for Machine Comprehension (2016, background): “A hybrid approach is to combine questions representations at multiple levels of granularity (unigrams, bigrams, trigrams) (Yang et al., 2015).”
Measuring and Improving BERT’s Mathematical Abilities by Predicting the Order of Reasoning. (2021, background): “…Hendricks et al., 2016), attention (Bahdanau et al., 2014; Mnih et al., 2014; Gulcehre et al., 2016; Malinowski et al., 2018; Xu and Saenko, 2016; Yang et al., 2016), phrase selection (Lei et al., 2016), distillation into programs (Hajipour et al., 2020), or decision trees (Alaniz and Akata,…”
Exploring Self-Attention for Image Recognition (2020, neutral): “This has inspired applications of self-attention and related ideas to image recognition [5, 34, 15, 14, 45, 46, 13, 1, 27], image synthesis [43, 26, 2], image captioning [39, 41, 4], and video prediction [17, 35].”
VisualBERT: A Simple and Performant Baseline for Vision and Language (2019, methodology): “…al., 2015; Goyal et al., 2017), textual grounding (Kazemzadeh et al., 2014; Plummer et al., 2015), and visual reasoning (Suhr et al., 2019; Zellers et al., 2019) have been proposed and various models (Yang et al., 2016; Anderson et al., 2018; Jiang et al., 2018) have been developed to solve them.”
Towards VQA Models That Can Read (2019, background): “VQA models typically use some variant of attention to get a representation of the image that is relevant for answering the given question [2, 7, 30, 47, 48, 51, 17].”
CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning (2016, methodology): “Following [44], these representations are combined using one or more rounds of soft spatial attention and the final answer distribution is predicted with an MLP.”
VOLO: Vision Outlooker for Visual Recognition (2021, neutral): “vision task [22], [29], [30], [31], [32], [33], [34], [35], [36], [37], [38], [39], [40], [41].”
How Much Can CLIP Benefit Vision-and-Language Tasks? (2021, methodology): “For training VQA models, we use hyperparameters of the open-source implementation9 from (Jiang et al., 2020) for the large version of the MCAN and base version of Pythia.”
AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks (2017, methodology): “It has been successfully used in modeling multi-level dependencies in image captioning [30, 38], image question answering [31] and machine translation [2].”
A simple neural network module for relational reasoning (2017, background): “eural networks. Next, both representations are combined, and the answers are either predicted or generated. Most successful methods also use an attention mechanism that locate important image regions [5, 44, 45, 46]. In our work, we follow a similar pipeline, but we use Relation Networks as a powerf”
Just Ask: Learning to Answer Questions from Millions of Narrated Videos (2020, neutral): “Various approaches to combine spatial image representations and sequential question representations have been proposed [7, 10, 25, 57, 86, 88, 91].”
Attention on Attention for Image Captioning (2019, background): “There are other attention mechanisms such as: spatial and channel-wise attention [6], adaptive attention [27], stacked attention [46], multi-level attention [51], multi-head attention and self-attention [35].”
OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge (2019, background): “Attention-based models [12, 31, 46, 47, 48, 54] better guide the model in answering the questions by highlighting image regions that are relevant to the question.”
Visual Dialog (2016, methodology): “Finally, we adapt several (near) state-of-art VQA models (SAN [59], HieCoAtt [30]) to Visual Dialog.”
LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention (2023, background): “Over the past decade, we have witnessed a shift in visionlanguage research from task-specific models [13,15,39,44, 52] to large foundation models [1, 4, 21, 36, 45–47, 50].”
Vision-and-Language Navigation: Interpreting Visually-Grounded Navigation Instructions in Real Environments (2017, background): “Although visual attention has also proved highly beneficial in vision and language problems [50, 27, 3], we leave an investigation of visual attention in Vision-and-Language Navigation to future work.”
Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering (2016, methodology): “uilding trust for machines among their users. 2. Related Work Visual Question Answering. A number of recent works have proposed visual question answering datasets [3,21,25, 30,10,45,37,35] and models [9,24,2,42,23,26,46, 44,43,40,34,19,28,14,41,32,16]. Our work builds on top of the VQA dataset from ”