Ask Your Neurons: A Deep Learning Approach to Visual Question Answering

By Mateusz Malinowski, Marcus Rohrbach, Mario Fritz. 2016

108 citations recorded by the source corpus.

We propose a Deep Learning approach to the visual question answering task, where machines answer to questions about real-world images. By combining latest advances in image representation and natural language processing, we propose Ask Your Neurons, a scalable, jointly trained, end-to-end formulation to this problem. In contrast to previous efforts, we are facing a multi-modal problem where the language output (answer) is conditioned on visual and natural language inputs (image and question). We evaluate our approaches on the DAQUAR as well as the VQA dataset where we also report various baselines, including an analysis how much information is contained in the language part only. To study human consensus, we propose two novel metrics and collect additional answers which extend the original DAQUAR dataset to DAQUAR-Consensus. Finally, we evaluate a rich set of design choices how to encode, combine and decode information in our proposed Deep Learning formulation.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

A simple neural network module for relational reasoning (2017, background): “They also contain ambiguities and exhibit strong linguistic biases that allow a model to learn answering strategies that exploit those biases, without reasoning about the visual input [1, 15, 19].”
Modulating early visual processing by language (2017, background): “For instance, it was shown that we can improve upon classic concatenation by an element-wise product [1, 13, 14], Tucker decomposition [2], bilinear pooling [9] or more exotic approaches [18].”
MUTAN: Multimodal Tucker Fusion for Visual Question Answering (2017, neutral): “e-art comparison results are gathered in Table 2. Firstly, we can notice that bilinear models, i.e. test-dev teststd Y/N No. Other All All SMem 2-hop [27] 80.87 37.32 43.12 57.99 58.24 Ask Your Neur. [20] 78.39 36.45 46.28 58.39 58.43 SAN [30] 79.3 36.6 46.1 58.7 58.9 D-NMN [1] 81.1 38.6 45.5 59.4 5”
Guide Me: Interacting with Deep Networks (2018, background): “Specialized systems for VQA ground the question in the input image and focus on the relevant parts to answer complex queries with text responses [4, 5, 27, 32, 43, 46].”
Learning Visual Question Answering by Bootstrapping Hard Attention (2018, methodology): “Answering questions about images is often formulated in terms of predictive models [24].”
Open-Ended Video Question Answering via Multi-Modal Conditional Adversarial Networks (2020, methodology): “state-of-the-art methods for the problem of open-ended video question answering as follows: • VQA+ method is the extension of image question algorithm [30], where the mean-pooling layer is added to encode the video.”
Learning to Navigate in Cities Without a Map (2018, background): “, 2016), CNN+LSTM (Donahue et al., 2015; Malinowski et al., 2017) architectures improve on this by exploiting spatio-temporal continuity (Weyand et al.”
Comprehensive Review of Artificial Neural Network Applications to Pattern Recognition (2019, background): “Also, over the years in research development, it is found that ‘‘accuracy of ANN in SR continues to be one key research challenges, for example, in the areas of speaking and language variability, speech domain, noise recognition, and vocabulary volume or size needed or used in speech [98]– [101].”
Spatiotemporal-Textual Co-Attention Network for Video Question Answering (2019, background): “VQA has attracted extensive attention in recent years [9, 29, 59], but it is still a challenging task relying on the accurate understanding of visual content and questions as well as the relevance between them [18, 33].”
Visual Grounding in Video for Unsupervised Word Translation (2020, methodology): “There is a growing interest in combining methods developed in computer vision and natural language processing to solve more challenging problems at the intersection of these fields [2, 13, 25, 27, 30, 33, 41, 45].”