Ask Your Neurons: A Neural-Based Approach to Answering Questions about Images

By Mateusz Malinowski, Marcus Rohrbach, Mario Fritz. 2015

637 citations recorded by the source corpus.

We address a question answering task on real-world images that is set up as a Visual Turing Test. By combining latest advances in image representation and natural language processing, we propose Neural-Image-QA, an end-to-end formulation to this problem for which all parts are trained jointly. In contrast to previous efforts, we are facing a multi-modal problem where the language output (answer) is conditioned on visual and natural language input (image and question). Our approach Neural-Image-QA doubles the performance of the previous best approach on this problem. We provide additional insights into the problem by analyzing how much information is contained only in the language part for which we provide a new human baseline. To study human consensus, which is related to the ambiguities inherent in this challenging task, we propose two novel metrics and collect additional answers which extends the original DAQUAR dataset to DAQUAR-Consensus.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

Bidirectional Attention Flow for Machine Comprehension (2016, methodology): “Early works on visual question answering (VQA) involved encoding the question using an RNN, encoding the image using a CNN and combining them to answer the question (Antol et al., 2015; Malinowski et al., 2015).”
HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips (2019, background): “These applications include automatic image and video captioning [20, 36, 60, 61], visual question answering [9, 29, 51, 63], visual content retrieval based on textual queries [32, 56, 63], temporal localization of events in videos using natural language [15, 26] or video summarization with natural l”
OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge (2019, background): “Early approaches to VQA combined recurrent networks with CNNs to integrate textual and visual data [27, 1].”
Visual Dialog (2016, background): “tion [55], object detection [27] – to ‘high-level’ AI tasks such as learning to play Atari video games [35] and Go [47], answering reading comprehension questions by understanding short stories [15, 57], and even answering questions about images [4, 32, 41] and videos [49, 50]!”
VQA: Visual Question Answering (2015, background): “In concurrent work, [31] proposed combining an LSTM for the question with a CNN for the image to generate an answer – a similar model is evaluated in this paper.”
Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering (2016, background): “A number of recent works have proposed visual question answering datasets [3, 22, 26, 31, 10, 46, 38, 36] and models [9, 25, 2, 43, 24, 27, 47, 45, 44, 41, 35, 20, 29, 15, 42, 33, 17].”
Stacked Attention Networks for Image Question Answering (2015, methodology): “The experimental results on DAQUAR-ALL, DAQUARREDUCED, COCO-QA and VQA are presented in Table.”
Hierarchical Question-Image Co-Attention for Visual Question Answering (2016, background): “1 Introduction Visual Question Answering (VQA) [2, 6, 14, 15, 27] has emerged as a prominent multi-discipline research problem in both academia and industry.”
Modulating early visual processing by language (2017, methodology): “The VQA task concerns answering open-ended questions about images and has received significant attention from the research community [1, 9, 17, 23].”
MovieQA: Understanding Stories in Movies through Question-Answering (2015, methodology): “In computer vision, [23] proposed a Bayesian approach on top of a logic-based QA system [20], while [24, 30] encoded both an image and the question using an LSTM and decoded an answer.”
Learning to Compose Neural Networks for Question Answering (2016, methodology): “…approaches to visual question answering either apply a recurrent model to deep representations of both the image and the question (Ren et al., 2015; Malinowski et al., 2015), or use the question to compute an attention over the input image, and then answer based on both the question and the image…”
Exploring Models and Data for Image Question Answering (2015, background): “This also makes the evaluation of the models easier and more robust, avoiding the thorny evaluation issues that plague multi-word generation problems.”
Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question (2015, methodology): “Different from [23, 32], the image representation does not feed into the LSTM in this component.”
TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering (2017, background): “Most existing VQA datasets are image-centric, e.g ., DAQUAR [25], abstract scenes [24], VQA [2], Visual Madlibs [41], DAQUAR-Consensus [26], FM-IQA [11], COCO-QA [29], and Visual7W [43].”