Human Attention in Visual Question Answering: Do Humans and Deep Networks look at the same regions?

By Abhishek Das, Harsh Agrawal, C. Lawrence Zitnick, Devi Parikh, Dhruv Batra. 2016

500 citations recorded by the source corpus.

We conduct large-scale studies on `human attention' in Visual Question Answering (VQA) to understand where humans choose to look to answer questions about images. We design and test multiple game-inspired novel attention-annotation interfaces that require the subject to sharpen regions of a blurred image to answer a question. Thus, we introduce the VQA-HAT (Human ATtention) dataset. We evaluate attention maps generated by state-of-the-art VQA models against human attention both qualitatively (via visualizations) and quantitatively (via rank-order correlation). Overall, our experiments show that current attention models in VQA do not seem to be looking at the same regions as humans.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Ming-Hsuan Yang
  4. Trevor Darrell
  5. Xiaogang Wang
  6. Jitendra Malik
  7. Cordelia Schmid
  8. Bernt Schiele

In Multimodal Learning

  1. Devi Parikh
  2. Mohit Bansal
  3. Zhe Gan
  4. Dhruv Batra
  5. Hanwang Zhang
  6. Marcus Rohrbach
  7. Trevor Darrell
  8. Linjie Li

In Question Answering & Reading Comprehension

  1. Christopher D. Manning
  2. Drew A. Hudson
  3. Mohit Bansal
  4. Nazneen Fatema Rajani and Raymond J. Mooney
  5. Christopher Kanan
  6. Devi Parikh
  7. Kushal Kafle
  8. Licheng Yu

Citations in the corpus