Exploring Models and Data for Image Question Answering

By Mengye Ren, Ryan Kiros, Richard S. Zemel. 2015

773 citations recorded by the source corpus.

This work aims to address the problem of image-based question-answering (QA) with new models and datasets. In our work, we propose to use neural networks and visual semantic embeddings, without intermediate stages such as object detection and image segmentation, to predict answers to simple questions about images. Our model performs 1.8 times better than the only published results on an existing image QA dataset. We also present a question generation algorithm that converts image descriptions, which are widely available, into QA form. We used this algorithm to produce an order-of-magnitude larger dataset, with more evenly distributed answers. A suite of baseline results on this new dataset are also presented.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

Towards VQA Models That Can Read (2019, methodology): “This dataset was larger, more natural, and more varied than earlier VQA datasets such as DAQUAR [31] or COCO-QA [38] but had linguistic priors which were exploited by models to answer questions without sufﬁcient visual grounding.”
Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations (2016, methodology): “Object detection is already a well-studied problem (Everingham et al., 2010, Girshick et al., 2014, Sermanet et al., 2013, Girshick, 2015, Ren et al., 2015b).”
Generative Adversarial Text to Image Synthesis (2016, methodology): “We mainly use the Caltech-UCSD Birds dataset and the Oxford-102 Flowers dataset along with ﬁve text descriptions per image we collected as our evaluation setting.”
Just Ask: Learning to Answer Questions from Millions of Narrated Videos (2020, neutral): “To address this issue, several works have explored leveraging manually annotated video descriptions [35, 82, 87, 96, 98, 99, 100] for automatic generation of VideoQA datasets, using rule-based [30, 66] approaches.”
OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge (2019, methodology): “In the past few years several datasets have been proposed for visual question answering [32, 3, 13, 51, 37, 54, 41, 27, 22, 44].”
VizWiz Grand Challenge: Answering Visual Questions from Blind People (2018, methodology): “Over the past three years, a plethora of VQA datasets have been publicly shared to encourage a larger community to collaborate on developing algorithms that answer visual questions [6, 7, 16, 17, 19, 20, 23, 28, 32, 39, 40, 43, 45].”
Visual Dialog (2016, background): “tion [55], object detection [27] – to ‘high-level’ AI tasks such as learning to play Atari video games [35] and Go [47], answering reading comprehension questions by understanding short stories [15, 57], and even answering questions about images [4, 32, 41] and videos [49, 50]!”
VQA: Visual Question Answering (2015, methodology): “[37] automatically generated four types of questions (object, count, color, location) using COCO captions.”
Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering (2016, background): “A number of recent works have proposed visual question answering datasets [3, 22, 26, 31, 10, 46, 38, 36] and models [9, 25, 2, 43, 24, 27, 47, 45, 44, 41, 35, 20, 29, 15, 42, 33, 17].”
Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models (2015, background): “l. 2015; Mao et al. 2015; Ordonez et al. 2011; Vinyals et al. 2015; Yao et al. 2010) and visual question answering (Antol et al. 2015; Gao et al. 2015; Krishna et al. 2016; Malinowski and Fritz 2014; Ren et al. 2015; Yu et al. 2015). Unfortunately, due to a lack of datasets that provide not only pai”
Neural Module Networks (2015, background): “We compare our approach to a reimplementation of the VIS+LSTM baseline similar to the one described by [33], again swapping out the pre-trained image embedding with a LeNet.”
Stacked Attention Networks for Image Question Answering (2015, methodology): “The experimental results on DAQUAR-ALL, DAQUARREDUCED, COCO-QA and VQA are presented in Table.”
Visual7W: Grounded Question Answering in Images (2015, methodology): “Several QA datasets [1, 7, 27, 36, 49] have been released since last year.”
Hierarchical Question-Image Co-Attention for Visual Question Answering (2016, methodology): “As in [15], we report classification accuracy as well as Wu-Palmer similarity (WUPS) in Table 2.”
Visual Entailment: A Novel Task for Fine-Grained Image Understanding (2019, methodology): “0 [23], CLEVR [32], Visual7w [81], Visual Genome [41], COCO-QA [57], and models [33, 60, 29, 31, 1, 67, 17, 37] have been used to measure the progress in understanding the interaction between vision and language modalities.”
TVQA: Localized, Compositional Video Question Answering (2018, methodology): “Visual Question Answering: Several image-based VQA datasets have recently been constructed, e.g., DAQUAR (Malinowski and Fritz, 2014), VQA (Antol et al., 2015), COCO-Q (Ren et al., 2015a), FM-IQA (Gao et al., 2015), Visual Madlibs (Yu et al., 2015), Visual7W (Zhu et al., 2016), CLEVR (Johnson et al.”
Modulating early visual processing by language (2017, methodology): “The majority of proposed methods use a similar computational pipeline introduced by [17, 20].”
MovieQA: Understanding Stories in Movies through Question-Answering (2015, methodology): “In computer vision, [23] proposed a Bayesian approach on top of a logic-based QA system [20], while [24, 30] encoded both an image and the question using an LSTM and decoded an answer.”
Learning to Compose Neural Networks for Question Answering (2016, background): “Most previous approaches to visual question answering either apply a recurrent model to deep representations of both the image and the question (Ren et al., 2015; Malinowski et al., 2015), or use the question to compute an attention over the input image, and then answer based on both the question an”