Hierarchical Question-Image Co-Attention for Visual Question Answering

By Jiasen Lu, Jianwei Yang, Dhruv Batra, Devi Parikh. 2016

1,744 citations recorded by the source corpus.

A number of recent works have proposed attention models for Visual Question Answering (VQA) that generate spatial maps highlighting image regions relevant to answering the question. In this paper, we argue that in addition to modeling "where to look" or visual attention, it is equally important to model "what words to listen to" or question attention. We present a novel co-attention model for VQA that jointly reasons about image and question attention. In addition, our model reasons about the question (and consequently the image via the co-attention mechanism) in a hierarchical fashion via a novel 1-dimensional convolution neural networks (CNN). Our model improves the state-of-the-art on the VQA dataset from 60.3% to 60.5%, and from 61.6% to 63.3% on the COCO-QA dataset. By using ResNet, the performance is further improved to 62.1% for VQA and 65.4% for COCO-QA.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

Bidirectional Attention Flow for Machine Comprehension (2016, background): “We conjecture that separating out the attention layer results in a richer set of features computed in the ﬁrst 4 layers which are then incorporated by the modeling layer.”
Large-Scale Adversarial Training for Vision-and-Language Representation Learning (2020, background): “Prominent work includes: (i) advanced attention mechanisms [40, 79, 51]; (ii) better multimodal fusion methods [14, 80, 27, 26]; (iii) multi-step reasoning [77, 22, 15]; (iv) incorporation of object relations [55, 48, 6, 32]; and (v) neural module networks for compositional reasoning [3, 25, 20, 11]”
LXMERT: Learning Cross-Modality Encoder Representations from Transformers (2019, background): “Lu et al. (2016) applies bi- directional attention to the vision-and-language tasks while its concurrent work BiDAF (Seo et al., 2017) adds modeling layers in solving reading comprehension.”
Towards VQA Models That Can Read (2019, neutral): “VQA models typically use some variant of attention to get a representation of the image that is relevant for answering the question [2, 7, 30, 47, 48, 51, 17].”
ShapeWorld - A new test methodology for multimodal language understanding (2017, methodology): “, yes/no questions), it neither requires the evaluated model to generate answers nor to rephrase the problem to fit it into a classification task of some sort – for instance, over the 1000/3000 most common answers, as is common practice recently (Lu et al., 2016; Fukui et al., 2016).”
Achieving Human Parity on Visual Question Answering (2021, background): “Traditional cross-modal fusionmethods gradually improve from a simple linear model [1] to bilinear pooling ones [20, 94], and attention-based fusion ones [52, 89], where questions and image features are fully interactedwith each other.”
NExT-QA: Next Phase of Question-Answering to Explaining Temporal Actions (2021, methodology): “Most of the improvements have been made in image QA [1, 2, 30] though video QA has received increasing attention recently.”
Decoupling the Role of Data, Attention, and Losses in Multimodal Transformers (2021, methodology): “To fine-tune for VQA, we replace the image–text matching loss with a 2-layer MLP and train with a binary cross-entropy loss against soft answer scores (Teney et al., 2018).”
Just Ask: Learning to Answer Questions from Millions of Narrated Videos (2020, neutral): “Various approaches to combine spatial image representations and sequential question representations have been proposed [7, 10, 25, 57, 86, 88, 91].”
OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge (2019, background): “Attention-based models [12, 31, 46, 47, 48, 54] better guide the model in answering the questions by highlighting image regions that are relevant to the question.”
Vision-and-Language Navigation: Interpreting Visually-Grounded Navigation Instructions in Real Environments (2017, background): “Although visual attention has also proved highly beneficial in vision and language problems [50, 27, 3], we leave an investigation of visual attention in Vision-and-Language Navigation to future work.”
FiLM: Visual Reasoning with a General Conditioning Layer (2017, background): “This approach differs from classical visual question answering pipelines which fuse image and language information into a single embedding via element-wise product, concatenation, attention, and/or more advanced methods (Yang et al. 2016; Lu et al. 2016; Anderson et al. 2017).”
Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering (2017, methodology): “Typically, these models can be characterized as top-down approaches, with context provided by a representation of a partially-completed caption in the case of image captioning [33, 27, 47, 45], or a representation of the question in the case of VQA [11, 28, 44, 46, 49].”
Dynamic Coattention Networks For Question Answering (2016, background): “Lu et al. (2016) proposed a hierarchical co-attention model for visual question answering, which achieved state of the art result on the COCO-VQA dataset (Antol et al., 2015).”
12-in-1: Multi-Task Vision and Language Representation Learning (2019, neutral): “Different from earlier work [16,22,28,29,50,51,55] which design bespoke architecture for different tasks, recently proposed models for V&L [1, 8, 23, 24, 27, 43, 45, 54] provide a common architecture that can be pretrained using self-supervised losses and adapted to many vision and language tasks.”
Machine Comprehension Using Match-LSTM and Answer Pointer (2016, methodology): “So the sequence order of the word-level attention representation used to align the figure and the question(Xu & Saenko, 2016; Fukui et al., 2016; Lu et al., 2016), are not used in VQA.”
Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding (2016, background): “Lu et al. (2016) recently proposed a model that extracts multiple co-attentions on the image and question and combines the co-attentions in a hierarchical manner using all of element-wise sums, concatenation, and fully connected layers.”