Generating Natural Questions About an Image
By Nasrin Mostafazadeh, Ishan Misra, Jacob Devlin, Margaret Mitchell, Xiaodong He, Lucy Vanderwende. 2016
315 citations recorded by the source corpus.
There has been an explosion of work in the vision & language community during the past few years from image captioning to video transcription, and answering questions about images. These tasks have focused on literal descriptions of the image. To move beyond the literal, we choose to explore how questions about an image are often directed at commonsense inference and the abstract events evoked by objects in the image. In this paper, we introduce the novel task of Visual Question Generation (VQG), where the system is tasked with asking a natural and engaging question when shown an image. We provide three datasets which cover a variety of images from object-centric to event-centric, with considerably more abstract training data than provided to state-of-the-art captioning systems thus far. We train and test several generative and retrieval models to tackle the task of VQG. Evaluation results show that while such models ask reasonable questions for a variety of images, there is still a wide gap with human performance which motivates further work on connecting images with commonsense knowledge and pragmatics. Our proposed task offers a new challenge to the community which we hope furthers interest in exploring deeper connections between vision & language.
Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Natural Language Generation
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Xiaogang Wang
- Dacheng Tao
- Trevor Darrell
- Bernt Schiele
- Ming-Hsuan Yang
- Jitendra Malik
- Andrew Zisserman
In Multimodal Learning
- Devi Parikh
- Jianfeng Gao
- Qi Wu
- Christopher Kanan
- Zhe Gan
- Dhruv Batra
- Kushal Kafle
- Anton van den Hengel
In Natural Language Generation
- Alexander Gerhard Schwing
- Idan Schwartz
- Sudha Rao
- Emiel Krahmer
- Jyoti Aneja
- Albert Gatt
- Unnat Jain
- Hal Daumé Iii
Citations in the corpus
- Just Ask: Learning to Answer Questions from Millions of Narrated Videos (2020, neutral): “Visual question generation (VQG) has been introduced in [59].”
- GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering (2019, background): “Neural alternatives to visual question generation have been recently proposed [25, 12, 35], but they aim at a quite different goal of creating engaging but potentially inaccurate questions about the wider context of the image such as subjective evoked feelings or speculative events that may lead to ”
- Learning to Ask: Neural Question Generation for Reading Comprehension (2017, methodology): “Mostafazadeh et al. (2016) introduce visual question generation task, to explore the deep connection between language and vision.”
- Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation (2017, methodology): “A further development is work on Visual Question Answering, where rather than descriptive captions, the aim is to produce responses to specific questions about images (Antol et al., 2015; Geman et al., 2015; Malinowski et al., 2016; Barnard, 2016; Mostafazadeh et al., 2016).”
- Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Models (2016, neutral): “We also report results on another novel task – Visual Question Generation (Mostafazadeh et al., 2016).”
- GQA: a new dataset for compositional question answering over real-world images (2019, background): “2 Neural alternatives to visual question generation have been recently proposed [29, 14, 42], but they aim at a quite different goal of creating “interesting and engaging” questions about the wider context of the image, e.”
- Identifying Well-formed Natural Language Questions (2018, background): “Identifying well-formed natural language questions can also facilitate a more nat-ural interaction between a user and a machine in personal assistants or chatbots (Yang et al., 2014; Mostafazadeh et al., 2016) or while recommending related queries in search-engines.”
- Interpretation of Natural Language Rules in Conversational Machine Reading (2018, background): “Mostafazadeh et al. (2016) investigate how to generate natural follow-up questions based on the content of an image.”
- Visual Question Generation as Dual Task of Visual Question Answering (2017, background): “Recently, two homogenous tasks, Visual Question Answering (VQA) [36, 34, 2, 19] and Visual Question Generation (VQG) [25, 35], have been introduced to the computer vision field as cross-modality learning tasks.”
- Attacking Visual Language Grounding with Adversarial Examples: A Case Study on Neural Image Captioning (2017, background): “…al., 2017), visual question answering (Antol et al., 2015; Fukui et al., 2016; Lu et al., 2016; Zhu et al., 2017), visual story-telling (Huang et al., 2016), natural question generation (Mostafazadeh et al., 2017, 2016), and image generation from captions (Mansimov et al., 2016; Reed et al., 2016).”
- G-DAug: Generative Data Augmentation for Commonsense Reasoning (2020, neutral): “, 2016) and visual question generation (Mostafazadeh et al., 2016).”
- Convolutional Image Captioning (2017, background): “LSTM nets have been considered as the de-facto standard for vision-language tasks of image captioning [5, 16, 37, 39, 38], visual question answering [3, 30, 28], question generation [14, 20], and visual dialog [7, 13], due to their compelling ability to memorize long-term depen-”
- Guided Transformer: Leveraging Multiple External Sources for Representation Learning in Conversational Search (2020, background): “he missing information. Hence, it is completely different form clarifying the user information needs in IR. Clarification has also studied in dialog systems and chat-bots [5, 17, 29], computer vision [31], and speech recognition [47]. However, since non of the above-mentioned systems are information”
- Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information (2018, background): “Mostafazadeh et al. (2016) introduce a Visual Question Generation task where the goal is to generate natural questions that are not about what is present in the image rather about what can be inferred given the image, somewhat analogous to clarification questions.”
- Semantic bottleneck for computer vision tasks (2018, result): “e generation of questions about images has driven less attention. We can, however, mention the interesting work of [38] where discriminative questions are produced to disambiguate pairs of images, or [39] which introduced the novel task (and dataset) of visual question generation. We can also mentio”
- What BERT Sees: Cross-Modal Transfer for Visual Question Generation (2020, methodology): “Suitable data for the VQG task can come from standard image datasets on which questions have been manually annotated, such as V QGCOCO, V QGFlickr, V QGBing (Mostafazadeh et al., 2016) , each consisting of 5000 images with 5 questions per image.”
- Quiz-Style Question Generation for News Stories (2021, unknown): “[12] However, many of these works have either assumed that (1) the answer is not needed or is provided as input, or (2) the question is asked in reference to a readily available source material, whether it be text, image, dialogue, or otherwise [31, 40, 41].”
- Two Can Play This Game: Visual Dialog with Discriminative Question Generation and Answering (2018, methodology): “[29] were the first to learn a question generation model using human-authored questions instead of machine-generated captions.”