ABC-CNN: An Attention Based Convolutional Neural Network for Visual Question Answering

By Kan Chen, Jiang Wang, Liang-Chieh Chen, Haoyuan Gao, Wei Xu, Ramkant Nevatia. 2015

297 citations recorded by the source corpus.

We propose a novel attention based deep learning architecture for visual question answering task (VQA). Given an image and an image related natural language question, VQA generates the natural language answer for the question. Generating the correct answers requires the model's attention to focus on the regions corresponding to the question, because different questions inquire about the attributes of different image regions. We introduce an attention based configurable convolutional neural network (ABC-CNN) to learn such question-guided attention. ABC-CNN determines an attention map for an image-question pair by convolving the image feature map with configurable convolutional kernels derived from the question's semantics. We evaluate the ABC-CNN architecture on three benchmark VQA datasets: Toronto COCO-QA, DAQUAR, and VQA dataset. ABC-CNN model achieves significant improvements over state-of-the-art methods on these datasets. The question-guided attention generated by ABC-CNN is also shown to reflect the regions that are highly relevant to the questions.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs (2016, methodology): “Interestingly, the atrous convolution technique has also been adopted for a broader set of tasks, such as object detection [12], [77], instance-level segmentation [78], visual question answering [79], and optical flow [80].”
Rationalizing Neural Predictions (2016, background): “Attention based models offer another lens to the inner workings of neural models (Bahdanau et al., 2015; Cheng et al., 2016; Martins and Astudillo, 2016; Chen et al., 2015; Xu and Saenko, 2015; Yang et al., 2015).”
Deep Modular Co-Attention Networks for Visual Question Answering (2019, background): “proposed a question-guided attention map that projected the question embeddings into the visual space and formulated a configurable convolutional kernel to search the image attention region [7].”
Analyzing the Behavior of Visual Question Answering Models (2016, background): “…in two ways – 1) the test question-image (QI) pair is “novel”, i.e., too different from training QI pairs; and 2) the test QI pair is “famil-iar”, but the answer required at test time is “novel”, i.e., answers seen during training are different from what needs to be produced for the test QI pairs.”
Attention to Scale: Scale-Aware Semantic Image Segmentation (2015, background): “The second type, share-net, applies multi-scale input images to a shared network.”
Simple Baseline for Visual Question Answering (2015, methodology): “Recently, several papers have appeared on arXiv (after CVPR’16 submission deadline) proposing neural network architectures for visual question answering, such as [13, 17, 5, 18, 16, 3, 11, 1].”
Stacking with Auxiliary Features for Visual Question Answering (2018, background): “A variety of methods to address these challenges have been developed in recent years (Fukui et al., 2016; Xu and Saenko, 2016; Lu et al., 2016; Chen et al., 2015).”
Multimodal Explanations: Justifying Decisions and Pointing to the Evidence (2018, methodology): “Initial approaches to VQA used full-frame representations [22], but most recent approaches use some form of spatial attention [36, 35, 40, 8, 34, 30, 14, 18].”
This looks like that: deep learning for interpretable image recognition (2018, background): “, 2015], visual question answering [Chen et al., 2015], and contour estimation [Xu et al.”
The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision (2019, background): “The state-of-the-art approaches usually use neural attentions (Malinowski & Fritz, 2014; Chen et al., 2015; Yang et al., 2016; Xu & Saenko, 2016).”
Tips and Tricks for Visual Question Answering: Learnings from the 2017 Challenge (2017, neutral): “[38, 34, 8, 19, 5, 35]).”
ABCNN: Attention-Based Convolutional Neural Network for Modeling Sentence Pairs (2015, background): “…is little if any work on attention mechanisms in CNNs for NLP, attention-based CNNs have been used in computer vision for visual question answering (Chen et al., 2015), image clas-siﬁcation (Xiao et al., 2015), caption generation (Xu et al., 2015), image segmentation (Hong et al., 2015) and object…”
Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering (2017, background): “has received a lot of attention – a number of VQA datasets have been curated [5, 21, 41, 12, 25, 11, 29, 13, 39] and a variety of deep-learning models have been developed [5, 9, 38, 37, 16, 3, 34, 19, 24, 4, 31, 20, 10, 26, 14, 35, 36, 40, 30].”
MarioQA: Answering Questions by Watching Gameplay Videos (2016, background): “On the other hand, image question answering (ImageQA) [3, 4, 5, 6, 10, 14, 22, 23, 24, 25, 28] addresses a holistic image understanding problem, and handles diverse recognition tasks in a single framework.”
Beyond Bilinear: Generalized Multimodal Factorized High-Order Pooling for Visual Question Answering (2017, background): “[21] proposed a questionguided attention map that projects the question embeddings to”
FVQA: Fact-Based Visual Question Answering (2016, background): “State-of-the-art RNN approaches [7], [10], [15], [17]–[20] directly learn the mapping between questions and answers, which, however, do not scale well to the diversity of answers and cannot provide the key information that the reasoning is based on.”
Multi-modal Factorized Bilinear Pooling with Co-attention Learning for Visual Question Answering (2017, background): “Chen et al. proposed a “question-guided attention map” that projects the question embeddings to the visual space and formulates a conﬁgurable convolutional kernel to search the image attention region [4].”
Understanding Convolution for Semantic Segmentation (2017, background): “More recently, dilated convolution has been applied to a broader range of tasks, such as object detection [26, 11], optical ﬂow [33], visual question answering [6], and audio generation [38].”
Learning Visual Question Answering by Bootstrapping Hard Attention (2018, background): “However, only soft attention is used in the majority of Visual QA works [7,8,9,10,11,12,43,44,45,46,47,48,49,50,51,52].”