Generating Natural Questions About an Image

By Nasrin Mostafazadeh, Ishan Misra, Jacob Devlin, Margaret Mitchell, Xiaodong He, Lucy Vanderwende. 2016

315 citations recorded by the source corpus.

There has been an explosion of work in the vision & language community during the past few years from image captioning to video transcription, and answering questions about images. These tasks have focused on literal descriptions of the image. To move beyond the literal, we choose to explore how questions about an image are often directed at commonsense inference and the abstract events evoked by objects in the image. In this paper, we introduce the novel task of Visual Question Generation (VQG), where the system is tasked with asking a natural and engaging question when shown an image. We provide three datasets which cover a variety of images from object-centric to event-centric, with considerably more abstract training data than provided to state-of-the-art captioning systems thus far. We train and test several generative and retrieval models to tackle the task of VQG. Evaluation results show that while such models ask reasonable questions for a variety of images, there is still a wide gap with human performance which motivates further work on connecting images with commonsense knowledge and pragmatics. Our proposed task offers a new challenge to the community which we hope furthers interest in exploring deeper connections between vision & language.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Natural Language Generation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Xiaogang Wang
  3. Dacheng Tao
  4. Trevor Darrell
  5. Bernt Schiele
  6. Ming-Hsuan Yang
  7. Jitendra Malik
  8. Andrew Zisserman

In Multimodal Learning

  1. Devi Parikh
  2. Jianfeng Gao
  3. Qi Wu
  4. Christopher Kanan
  5. Zhe Gan
  6. Dhruv Batra
  7. Kushal Kafle
  8. Anton van den Hengel

In Natural Language Generation

  1. Alexander Gerhard Schwing
  2. Idan Schwartz
  3. Sudha Rao
  4. Emiel Krahmer
  5. Jyoti Aneja
  6. Albert Gatt
  7. Unnat Jain
  8. Hal Daumé Iii

Citations in the corpus