A Multi-World Approach to Question Answering about Real-World Scenes based on Uncertain Input

By Mateusz Malinowski, Mario Fritz. 2014

747 citations recorded by the source corpus.

We propose a method for automatically answering questions about images by bringing together recent advances from natural language processing and computer vision. We combine discrete reasoning with uncertain predictions by a multi-world approach that represents uncertainty about the perceived world in a bayesian framework. Our approach can handle human questions of high complexity about realistic scenes and replies with range of answer like counts, object classes, instances and lists of them. The system is directly trained from question-answer pairs. We establish a first benchmark for this task that can be seen as a modern attempt at a visual turing test.

Subfields: Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books (2015, background): “The field has tackled a diverse set of tasks such as captioning [15, 13, 40, 39, 24], alignment [13, 17, 38], Q&A [22, 21], visual model learning from textual descriptions [9, 29], and semantic visual search with natural multisentence queries [19].”
Measuring and Improving BERT’s Mathematical Abilities by Predicting the Order of Reasoning. (2021, background): “Symbolic or semi-symbolic architectures naturally produce justiﬁcations as a sequence of formulas in some formal language (Lane et al., 2005; Core et al., 2006; Lomas et al., 2012; Johnson; Liang, 2016; Malinowski and Fritz, 2014).”
Towards VQA Models That Can Read (2019, methodology): “This dataset was larger, more natural, and more varied than earlier VQA datasets such as DAQUAR [31] or COCO-QA [38] but had linguistic priors which were exploited by models to answer questions without sufﬁcient visual grounding.”
CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning (2016, background): “Recently, several datasets have been introduced to study this problem [4, 10, 21, 26, 32, 46, 49].”
Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations (2016, methodology): “The proposed models range from SVM classifiers (Antol et al., 2015) and probabilistic inference (Malinowski and Fritz, 2014) to recurrent neural networks (Gao et al., 2015, Malinowski et al., 2015, Ren et al., 2015a) and convolutional networks (Ma et al., 2015).”
A simple neural network module for relational reasoning (2017, background): “This is a challenging problem domain because it requires high-level scene understanding [1, 14].”
NExT-QA: Next Phase of Question-Answering to Explaining Temporal Actions (2021, methodology): “Then, we determine the Wu-Palmer similarity (WUPS) score 3 [32] to evaluate the quality of the generated answers.”
OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge (2019, methodology): “We can see from the average question lengths and average answer lengths that our questions and answers are about comparable to KB-VQA [43] and FVQA [44] and longer than the other VQA datasets with the exception of DAQUAR and CLEVR (which are partially and fully automated from templates respectively)”
VizWiz Grand Challenge: Answering Visual Questions from Blind People (2018, methodology): “Over the past three years, a plethora of VQA datasets have been publicly shared to encourage a larger community to collaborate on developing algorithms that answer visual questions [6, 7, 16, 17, 19, 20, 23, 28, 32, 39, 40, 43, 45].”
Visual Dialog (2016, background): “A number of problems at the intersection of vision and language have recently gained prominence – image captioning [10, 11, 20, 54], video/movie description [43, 51, 52], text-to-image coreference/grounding [16, 22, 37, 39, 42], Visual Madlibs [60], and of course, visual question answering (VQA) [4,”
VQA: Visual Question Answering (2015, background): “For instance, [30] only considers questions whose answers come from a predefined closed world of 16 basic colors or 894 object categories.”
FiLM: Visual Reasoning with a General Conditioning Layer (2017, background): “Visual question answering itself has its own line of datasets (Malinowski and Fritz 2014; Geman et al. 2015; Antol et al. 2015) which focus on asking a diverse set of simpler questions on images, often answerable in a single glance.”
Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models (2015, background): “…et al. 2014; Klein et al. 2014; Kulkarni et al. 2011; Lebret et al. 2015; Mao et al. 2015b; Ordonez et al. 2011; Vinyals et al. 2015; Yao et al. 2010) and visual question answering (Antol et al. 2015; Gao et al. 2015; Krishna et al. 2016; Malinowski and Fritz 2014; Ren et al. 2015; Yu et al. 2015).”
GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering (2019, methodology): “Contrary to GQA, those datasets are either small in scale [22] or use only a restricted set of objects and a handful of non-compositional templates [15, 21].”
Neural Module Networks (2015, methodology): “Notable “classical” approaches to this task include [27, 21].”
Stacked Attention Networks for Image Question Answering (2015, methodology): “We compare our models with a set of baselines proposed recently [21, 1, 18, 19, 17] on image QA.”
Ask Me Anything: Dynamic Memory Networks for Natural Language Processing (2015, background): “…have recently been proposed for algorithmic problems (Joulin & Mikolov, 2015; Kaiser & Sutskever, 2015), caption generation for images (Malinowski & Fritz, 2014; Chen & Zitnick, 2014), visual question answering (Yang et al., 2015) or other NLP problems and datasets (Hermann et al.,…”
Visual7W: Grounded Question Answering in Images (2015, background): “The DAQUAR dataset is the ﬁrst toy-sized QA benchmark built upon indoor scene RGB-D images.”
Visual Entailment: A Novel Task for Fine-Grained Image Understanding (2019, neutral): “Recent work on VQA includes datasets [32, 2, 23, 81, 41, 57, 47, 19, 66] and models [33, 60, 29, 31, 1, 67, 17, 37].”