Ask Me Anything: Free-Form Visual Question Answering Based on Knowledge from External Sources

By Qi Wu, Peng Wang, Chunhua Shen, Anthony R. Dick, Anton van den Hengel. 2015

392 citations recorded by the source corpus.

We propose a method for visual question answering which combines an internal representation of the content of an image with information extracted from a general knowledge base to answer a broad range of image-based questions. This allows more complex questions to be answered using the predominant neural network-based approach than has previously been possible. It particularly allows questions to be asked about the contents of an image, even when the image itself does not contain the whole answer. The method constructs a textual representation of the semantic content of an image, and merges it with textual information sourced from a knowledge base, to develop a deeper understanding of the scene viewed. Priming a recurrent neural network with this combined information, and the submitted question, leads to a very flexible visual question answering approach. We are specifically able to answer questions posed in natural language, that refer to information not contained in the image. We demonstrate the effectiveness of our model on two publicly available datasets, Toronto COCO-QA [23] and VQA [1] and show that it produces the best reported results in both cases.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Retrieval-Augmented Generation, Multimodal Learning, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Jianfeng Gao
  3. Dacheng Tao
  4. Trevor Darrell
  5. Yoshua Bengio
  6. Devi Parikh
  7. Yejin Choi
  8. Percy Liang

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Trevor Darrell
  4. Xiaogang Wang
  5. Yi Yang
  6. Ming-Hsuan Yang
  7. Jitendra Malik
  8. Cordelia Schmid

In Retrieval-Augmented Generation

  1. Anton van den Hengel
  2. Christopher Kanan
  3. Kushal Kafle
  4. Damien Teney
  5. Chunhua Shen
  6. Qi Wu
  7. Peng Wang
  8. Anthony R. Dick

In Multimodal Learning

  1. Devi Parikh
  2. Anton van den Hengel
  3. Christopher Kanan
  4. Qi Wu
  5. Kushal Kafle
  6. Marcus Rohrbach
  7. Dhruv Batra
  8. Mohit Bansal

In Question Answering & Reading Comprehension

  1. Christopher Kanan
  2. Anton van den Hengel
  3. Kushal Kafle
  4. Damien Teney
  5. Devi Parikh
  6. Qi Wu
  7. Yueting Zhuang
  8. Hanwang Zhang

Citations in the corpus