Measuring Machine Intelligence Through Visual Question Answering

By C. Lawrence Zitnick, Aishwarya Agrawal, Stanislaw Antol, Margaret Mitchell, Dhruv Batra, Devi Parikh. 2016

38 citations recorded by the source corpus.

As machines have become more intelligent, there has been a renewed interest in methods for measuring their intelligence. A common approach is to propose tasks for which a human excels, but one which machines find difficult. However, an ideal task should also be easy to evaluate and not be easily gameable. We begin with a case study exploring the recently popular task of image captioning and its limitations as a task for measuring machine intelligence. An alternative and more promising task is Visual Question Answering that tests a machine’s ability to reason about language and vision. We describe a dataset unprecedented in size created for the task that contains over 760,000 human generated questions about images. Using around 10 million human generated answers, machines may be easily evaluated.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Question Answering & Reading Comprehension

Citations in the corpus

Visual Dialog (2016, background): “such as learning to play Atari video games [36] and Go [49], answering reading comprehension questions by understanding short stories [18, 59], and even answering questions about images [4, 34, 43, 64] and videos [51, 52]!”
Hierarchical Question-Image Co-Attention for Visual Question Answering (2016, background): “1 Introduction Visual Question Answering (VQA) [2, 6, 14, 15, 27] has emerged as a prominent multi-discipline research problem in both academia and industry.”
Multimodal Explanations: Justifying Decisions and Pointing to the Evidence (2018, background): “To provide textual explanations for questions that go beyond such trivial cases, we consider the annotations collected in [42] which say how old a human must be to answer a question.”
The Promise of Premise: Harnessing Question Premises in Visual Question Answering (2017, background): “…an image – i.e. Visual Question Answering (VQA) – has received substantial attention in the past few years (Malinowski and Fritz, 2014; Antol et al., 2015; Malinowski et al., 2015; Zitnick et al., 2016; Kim et al., 2016; Wu et al., 2016; Lu et al., 2016; Andreas et al., ∗Denotes equal contribution.”
Straight to the Facts: Learning Knowledge Base Retrieval for Factual Visual Question Answering (2018, methodology): “Most of the methods developed for visual question answering [2,8,10,18,19,20,21,22,23,24,12,27,29,30,31,6,7,32,33,34] focus exclusively on answering questions related to observed content.”
Multi-level Attention Networks for Visual Question Answering (2017, background): “Visual question answering (VQA) has attracted extensive attention recently, since VQA is considered approaching towards the milestone of “AI-complete” that enables a machine to reason across language and vision as humans [38].”
VSE++: Improving Visual-Semantic Embeddings with Hard Negatives (2017, background): “One approach to visual question-answering, for example, is to ﬁrst describe an image by a set of captions, and then to ﬁnd the nearest caption in response to a question ([1, 37]).”
Creativity: Generating Diverse Questions Using Variational Autoencoders (2017, background): “When considering generation of text from images, caption and paragraph generation [32, 4, 11, 18, 19, 20, 13, 33, 37, 40, 48, 60, 64, 69], as well as visual question answering [2, 22, 47, 52, 57, 67, 68, 71, 21, 34, 74, 1, 15, 31, 72, 73, 66, 46, 45, 31] come to mind immediately.”