Simple Baseline for Visual Question Answering

By Bolei Zhou, Yuandong Tian, Sainbayar Sukhbaatar, Arthur Szlam, Rob Fergus. 2015

339 citations recorded by the source corpus.

We describe a very simple bag-of-words baseline for visual question answering. This baseline concatenates the word features from the question and CNN features from the image to predict the answer. When evaluated on the challenging VQA dataset [2], it shows comparable performance to many recent approaches using recurrent neural networks. To explore the strength and weakness of the trained model, we also provide an interactive web demo and open-source code. .

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes (2020, neutral): “Similar problems were found in VQA [3], where simple baselines without sophisticated multimodal understanding performed remarkably well [94, 35, 1, 26] and in multimodal machine translation [18, 74], where the image was found to matter relatively little [13, 17, 7].”
Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering (2016, background): “lt in good superﬁcial performance, without the underlying models truly understanding the visual content. This phenomenon has been observed in image captioning [6] as well as visual question answering [47,49,16,18, 1]. For instance, in the VQA [3] dataset, the most common sport answer “tennis” is the”
Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding (2016, methodology): “Simpler models such as iBOWIMG baseline (Zhou et al., 2015) use concatenation and fully connected layers to combine the image and question modalities.”
Learning to Reason: End-to-End Module Networks for Visual Question Answering (2017, neutral): “ventional linguistic analysis of the question. Compare Integer Query Attribute Compare Attribute Method Overall Exist Count equal less more size color material shape size color material shape CNN+BoW [26] 48.4 59.5 38.9 50 54 49 56 32 58 47 52 52 51 52 CNN+LSTM [4] 52.3 65.2 43.7 57 72 69 59 32 58 4”
The Effect of Different Writing Tasks on Linguistic Style: A Case Study of the ROC Story Cloze Task (2017, background): “Similar problems have been shown in visual question answering datasets, where simple models that rely mostly on the question text perform competitively with state of the art models by exploiting language biases (Zhou et al., 2015; Jabri et al., 2016).”
Learning to Compose Neural Networks for Question Answering (2016, methodology): “We achieve state-of-theart results on this task, outperforming a highly effective visual bag-of-words model (Zhou et al., 2015), a model with dynamic network parameter prediction (but fixed network structure) (Noh et al.”
Deep Modular Co-Attention Networks for Visual Question Answering (2019, background): “The image and question are first represented as global features and then fused by a multimodal fusion model to predict the answer [37].”
Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering (2015, methodology): “For the VQA dataset, we use the simple iBOWIMG model in [3] as one baseline model, which beats most existing VQA models currently on arxiv.org.”
Dynamic Memory Networks for Visual and Textual Question Answering (2016, methodology): “End-to-end memory networks (Sukhbaatar et al., 2015) do not have this limitation.”
Learning Deep Features for Discriminative Localization (2015, positive): “Furthermore we demonstrate that the CAM localization technique generalizes to other visual recognition tasks i.e., our technique produces generic localizable deep features that can aid other researchers in understanding the basis of discrimination used by CNNs for their tasks.”
Stacking with Auxiliary Features for Visual Question Answering (2018, background): “We also compare against other state-of-the-art VQA systems not used in our ensemble: iBowIMG (Zhou et al., 2015), DPPNet”
Dual Attention Networks for Multimodal Reasoning and Matching (2016, background): “Recently, dramatic advances in deep learning have broken the boundaries between vision and language, drawing growing interest in their intersection, such as visual question answering (VQA) [3, 37, 23, 35], image captioning [33, 2], image-text matching [8, 11, 20, 30], visual grounding [24, 9], etc.”
Where to Look: Focus Regions for Visual Question Answering (2015, neutral): “7).”
Dynamic Fusion With Intra- and Inter-Modality Attention Flow for Visual Question Answering (2018, methodology): “In early VQA methods, simple concatenation or element-wise multiplication [45] between visual and language are used for cross-modal feature fusion.”
MarioQA: Answering Questions by Watching Gameplay Videos (2016, background): “On the other hand, image question answering (ImageQA) [3, 4, 5, 6, 10, 14, 22, 23, 24, 25, 28] addresses a holistic image understanding problem, and handles diverse recognition tasks in a single framework.”
Visual question answering: Datasets, algorithms, and future challenges (2016, methodology): “In [38], the authors used a bag-of-words to represent the question and CNN features from GoogLeNet for the visual features.”
An Analysis of Visual Question Answering Algorithms (2017, methodology): “We only use Visual Genome to make color and positional reasoning questions.”