Towards a Visual Turing Challenge

By Mateusz Malinowski, Mario Fritz. 2014

74 citations recorded by the source corpus.

As language and visual understanding by machines progresses rapidly, we are observing an increasing interest in holistic architectures that tightly interlink both modalities in a joint learning and inference process. This trend has allowed the community to progress towards more challenging and open tasks and refueled the hope at achieving the old AI dream of building machines that could pass a turing test in open domains. In order to steadily make progress towards this goal, we realize that quantifying performance becomes increasingly difficult. Therefore we ask how we can precisely define such challenges and how we can evaluate different algorithms on this open tasks? In this paper, we summarize and discuss such challenges as well as try to give answers where appropriate options are available in the literature. We exemplify some of the solutions on a recently presented dataset of question-answering task based on real-world indoor images that establishes a visual turing challenge. Finally, we argue despite the success of unique ground-truth annotation, we likely have to step away from carefully curated dataset and rather rely on 'social consensus' as the main driving force to create suitable benchmarks. Providing coverage in this inherently ambiguous output space is an emerging challenge that we face in order to make quantifiable progress in this area.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

Exploring Models and Data for Image Question Answering (2015, methodology): “To evaluate model performance, we used the plain answer accuracy as well as the Wu-Palmer similarity (WUPS) measure (Wu & Palmer, 1994; Malinowski & Fritz, 2014b).”
Ask Your Neurons: A Neural-Based Approach to Answering Questions about Images (2015, methodology): “Most recently several approaches have been proposed to approach Visual Turing Test [21], i.”
MUTAN: Multimodal Tucker Fusion for Visual Question Answering (2017, background): “Instead of explicitly learning an alignment between two spaces, the goal of VQA [2, 19] is to merge both modalities in order to decide which answer is correct.”
Ask Me Anything: Free-Form Visual Question Answering Based on Knowledge from External Sources (2015, methodology): “We did not test on the DAQUAR dataset [19] as it is an order of magnitude smaller than the datasets mentioned above, and thus too small to train our system, and to test its generality.”
Tips and Tricks for Visual Question Answering: Learnings from the 2017 Challenge (2017, background): “[6], mainly from the Computer Vision community as VQA constitutes a practical setting to evaluate deep visual understanding [25, 26].”
Explicit Knowledge-based Reasoning for Visual Question Answering (2015, methodology): “Several VQA datasets [Malinowski and Fritz, 2014b; Ren et al., 2015; Antol et al., 2015; Yu et al., 2015; Krishna et al., 2017] have also been constructed for making and measuring progress.”
Visual question answering: Datasets, algorithms, and future challenges (2016, methodology): “A Visual Turing Test rigorously evaluates a computer vision system to assess whether it is capable of human-level semantic analysis of images [8, 9].”
FVQA: Fact-Based Visual Question Answering (2016, methodology): “able large-scale data for training neural-network based VQA models and introduced various question types, and tasks, from global association between QA sentences and images [5,6,9] to grounded QA in image regions [10]; from free-from answer generation [5, 7, 9, 10] to multiple-choice picking [5, 6] ”
Automatic Description Generation from Images: A Survey of Models, Datasets, and Evaluation Measures (2016, methodology): “Following this effort, several datasets on this task are being released: DAQUAR (Malinowski & Fritz, 2014a) was compiled from scene depth images and mainly focuses on questions about the type, quantity and color of objects; COCOQA (Ren et al., 2015) was constructed by converting image descriptions t”
Semantically-enriched 3D models for common-sense knowledge (2015, background): “Most recently, the vision community is focusing on defining a Visual Turing test for deep understanding of visual structure and semantics in order to perform complex queries over image datasets for question answering and image retrieval tasks [9, 17].”
Visual Question Generation as Dual Task of Visual Question Answering (2017, background): “Additionally, several benchmark datasets are proposed to facilitate the VQA research [20, 2].”
Learning Visual Question Answering by Bootstrapping Hard Attention (2018, neutral): “[6,23,29,30,31,32].”
Ask Your Neurons: A Deep Learning Approach to Visual Question Answering (2016, methodology): “ been applied to question answering about images, for instance byYang et al(2015). 2.6 Visual Turing Test Recently, a large number architectures have been proposed to approach the Visual Turing Test (Malinowski and Fritz, 2014c), frequently also referred to as “Visual Question Answering”. They range”
Compact Trilinear Interaction for Visual Question Answering (2019, background): “The aim of VQA is to find out a correct answer for a given question which is consistent with visual content of a given image [25, 3, 10].”
KRISP: Integrating Implicit and Symbolic Knowledge for Open-Domain Knowledge-Based VQA (2020, background): “Early work in VQA focused on image and question parsing [2, 6, 23, 49, 50] assuming all required knowledge can be learned from the VQA training set.”
On transfer learning using a MAC model variant (2018, methodology): “Most of the VQA datasets have strong biases.”
Image Captioning and Visual Question Answering Based on Attributes and External Knowledge (2016, unknown): “DAQURA-ALL is proposed in [78].”
Straight to the Facts: Learning Knowledge Base Retrieval for Factual Visual Question Answering (2018, background): “For instance, signiﬁcant progress in VQA was achieved by introducing a variety of VQA datasets with strong baselines [1,2,3,4,5,6,7,8].”