Image Question Answering: A Visual Semantic Embedding Model and a New Dataset

By Mengye Ren, Ryan Kiros, Richard S. Zemel. 2015

153 citations recorded by the source corpus.

Subfields: Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

ShapeWorld - A new test methodology for multimodal language understanding (2017, methodology): “We evaluate several multimodal deep neural network architectures that were recently proposed for VQA (Goyal et al., 2016; Agrawal et al., 2016; Jabri et al., 2016; Antol et al., 2015; Ren et al., 2015).”
A simple neural network module for relational reasoning (2017, background): “he newly introduced CLEVR dataset [15] that distills core challenges of the task, namely relational and multi-modal reasoning. The majority of approaches to question answering share the same pipeline [6, 31, 36]. First, questions are encoded with recurrent neural networks, and images are encoded wit”
Neural Module Networks (2015, background): “uch) (Figure 1). The visual QA task has signiﬁcant signiﬁcant applications to human-robot interaction, search, and accessibility, and has been the subject of a great deal of recent research attention [2,7,20,22,25,32]. The task requires sophisticated understanding of both visual scenes and natural l”
Learning to Compose Neural Networks for Question Answering (2016, background): “Most previous approaches to visual question answering either apply a recurrent model to deep representations of both the image and the question (Ren et al., 2015; Malinowski et al., 2015), or use the question to compute an attention over the input image, and then answer based on both the question an”
Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question (2015, methodology): “Different from [23, 32], the image representation does not feed into the LSTM in this component.”
Natural Language Object Retrieval (2015, methodology): “Compared with other types of visual-linguistic models such as bag-of-words [26], one of the advantages of using a recurrent neural network as scoring function is that the whole model can be easily learned end-to-end via simple back propagation, allowing visual feature extraction and text sequence em”
Learning to Answer Questions from Image Using Convolutional Neural Network (2015, methodology): “The previous work (Malinowski et al., 2015b; Ren et al., 2015) introduced the image representation into LSTM, which are of worse performances than our CNN model.”
Ask Me Anything: Free-Form Visual Question Answering Based on Knowledge from External Sources (2015, methodology): “In addition, it achieves an accuracy of 69.73% on the Toronto COCO-QA, while the latest state-of-the-art is 55.92%.”
Accelerating Very Deep Convolutional Networks for Classification and Detection (2015, neutral): “detection [2], [9], [10], [11], semantic segmentation [12], [13], [14], [40], [41], image captioning [42], [43], [44], video/action recognition [45], image question answering [46], texture recognition [47], etc.”
FVQA: Fact-Based Visual Question Answering (2016, methodology): “In demonstrating the value of the dataset in driving deeper levels of image understanding in VQA we examine the performance of the state-of-the-art LSTM (Long-Short Term Memory) models [5, 6, 9] on our FVQA dataset.”
Neural Motifs: Scene Graph Parsing with Global Context (2017, background): “Deep sequential models have demonstrated strong performance for tasks such as captioning [6, 9, 40, 18] and visual question answer [1, 34, 48, 12, 8], including for problems not classically thought of as sequence problems, such as language parsing [39], or multi-label classiﬁcation [41].”
Learning Visual Question Answering by Bootstrapping Hard Attention (2018, methodology): “We compute a combined representation by copying the question representation to every spatial location in the CNN, and concatenating it with (or simply adding it to) the visual features, like previous Otherwise, we follow the canonical Visual QA pipeline [7,9,22,23,24,25].”
Ask Your Neurons: A Deep Learning Approach to Visual Question Answering (2016, methodology): “Most contemporary approaches use a global image representation, i.e. they encode the whole image with a CNN. Questions are then encoded with an RNN (Malinowski et al, 2015; Ren et al, 2015; Gao et al, 2015) or a CNN (Ma et al, 2016).”
Visual question answering: A survey of methods and datasets (2016, methodology): “The difference with COCO-QA is that the questions/answers are provided here by humans through the Amazon Mechanical Turk crowd-sourcing platform.”
PlotQA: Reasoning over Scientific Plots (2019, methodology): “These include datasets such as COCO-QA [28], DAQUAR [23], VQA [1, 7] which contain questions asked over natu-ral images.”