Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question

By Haoyuan Gao, Junhua Mao, Jie Zhou, Zhiheng Huang, Lei Wang, Wei Xu. 2015

531 citations recorded by the source corpus.

In this paper, we present the mQA model, which is able to answer questions about the content of an image. The answer can be a sentence, a phrase or a single word. Our model contains four components: a Long Short-Term Memory (LSTM) to extract the question representation, a Convolutional Neural Network (CNN) to extract the visual representation, an LSTM for storing the linguistic context in an answer, and a fusing component to combine the information from the first three components and generate the answer. We construct a Freestyle Multilingual Image Question Answering (FM-IQA) dataset to train and evaluate our mQA model. It contains over 150,000 images and 310,000 freestyle Chinese question-answer pairs and their English translations. The quality of the generated answers of our mQA model on this dataset is evaluated by human judges through a Turing Test. Specifically, we mix the answers provided by humans and our model. The human judges need to distinguish our model from the human. They will also provide a score (i.e. 0, 1, 2, the larger the better) indicating the quality of the answer. We propose strategies to monitor the quality of this evaluation process. The experiments show that in 64.7% of cases, the human judges cannot distinguish our model from humans. The average score is 1.454 (1.918 for human). The details of this work, including the FM-IQA dataset, can be found on the project page: this http URL

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages (2020, methodology): “2018), ARCD—an Arabic reading comprehension dataset (Mozannar et al., 2019), a HindiEnglish parallel dataset in a SQuAD-like setting (Guptaetal.,2018),andaChinese-Englishdataset focused on visual QA (Gao et al., 2015). The recent MLQA and XQuAD datasets also translate SQuAD in several languages (see”
A simple neural network module for relational reasoning (2017, background): “he newly introduced CLEVR dataset [15] that distills core challenges of the task, namely relational and multi-modal reasoning. The majority of approaches to question answering share the same pipeline [6, 31, 36]. First, questions are encoded with recurrent neural networks, and images are encoded wit”
OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge (2019, background): “FM-IQA [13] is another dataset that includes multi-lingual questions and answers.”
Visual Dialog (2016, background): “One key difference between VisDial and previous image question-answering datasets (VQA [6], Visual 7W [70], Baidu mQA [17]) is the lack of a ‘vi-sual priming bias’ in VisDial.”
VQA: Visual Question Answering (2015, background): “Concurrent with our work, [15] collected questions & answers in Chinese (later translated to English) for COCO images.”
Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering (2016, background): “A number of recent works have proposed visual question answering datasets [3, 22, 26, 31, 10, 46, 38, 36] and models [9, 25, 2, 43, 24, 27, 47, 45, 44, 41, 35, 20, 29, 15, 42, 33, 17].”
Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models (2015, background): “…et al. 2014; Klein et al. 2014; Kulkarni et al. 2011; Lebret et al. 2015; Mao et al. 2015b; Ordonez et al. 2011; Vinyals et al. 2015; Yao et al. 2010) and visual question answering (Antol et al. 2015; Gao et al. 2015; Krishna et al. 2016; Malinowski and Fritz 2014; Ren et al. 2015; Yu et al. 2015).”
Neural Module Networks (2015, background): “Several neural models for visual questioning have already been proposed in the literature [33, 26, 10], all of which use standard deep sequence modeling machinery to construct a joint embedding of image and text, which is immediately mapped to a distribution over answers.”
Stacked Attention Networks for Image Question Answering (2015, methodology): “[7] created an image QA data set through human labeling.”
Learning Deep Structure-Preserving Image-Text Embeddings (2015, background): “There is a surge of interest in image-text tasks such as image captioning [10, 22, 23, 25, 31, 43, 46, 50] and visual question answering [2, 12, 52].”
Visual7W: Grounded Question Answering in Images (2015, methodology): “Inspired by Geman et al.’s prototype of a visual Turing test based on image regions [8] and the comprehensive data collection of QA pairs on COCO images [25] such as VQA [1] and Baidu [7], we fuse visual QA and grounding in order to create a new QA dataset with dense annotations and a more ﬂexible e”
Hierarchical Question-Image Co-Attention for Visual Question Answering (2016, background): “1 Introduction Visual Question Answering (VQA) [2, 6, 14, 15, 27] has emerged as a prominent multi-discipline research problem in both academia and industry.”
Visual Entailment: A Novel Task for Fine-Grained Image Understanding (2019, neutral): “Recent work on VQA includes datasets [32, 2, 23, 81, 41, 57, 47, 19, 66] and models [33, 60, 29, 31, 1, 67, 17, 37].”
TVQA: Localized, Compositional Video Question Answering (2018, background): “, 2015a), FM-IQA (Gao et al., 2015), Visual Madlibs (Yu et al.”
Exploring Models and Data for Image Question Answering (2015, methodology): “Very recently there has been a number of parallel efforts on both creating datasets and proposing new models (Antol et al., 2015; Malinowski et al., 2015; Gao et al., 2015; Ma et al., 2015).”