TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering

By Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, Gunhee Kim. 2017

700 citations recorded by the source corpus.

Vision and language understanding has emerged as a subject undergoing intense study in Artificial Intelligence. Among many tasks in this line of research, visual question answering (VQA) has been one of the most successful ones, where the goal is to learn a model that understands visual content at region-level details and finds their associations with pairs of questions and answers in the natural language form. Despite the rapid progress in the past few years, most existing work in VQA have focused primarily on images. In this paper, we focus on extending VQA to the video domain and contribute to the literature in three important ways. First, we propose three new tasks designed specifically for video VQA, which require spatio-temporal reasoning from videos to answer questions correctly. Next, we introduce a new large-scale dataset for video VQA named TGIF-QA that extends existing VQA work with our new tasks. Finally, we propose a dual-LSTM based approach with both spatial and temporal attention, and show its effectiveness over conventional VQA techniques through empirical evaluations.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Jianfeng Gao
  3. Zhe Gan
  4. Yejin Choi
  5. Dacheng Tao
  6. Percy Liang
  7. Cordelia Schmid
  8. Caiming Xiong

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Cordelia Schmid
  4. Trevor Darrell
  5. Xiaogang Wang
  6. Yi Yang
  7. Ming-Hsuan Yang
  8. Jitendra Malik

In Question Answering & Reading Comprehension

  1. Chenyou Fan
  2. Mohit Bansal
  3. Tat-seng Chua
  4. Jie Lei
  5. Junbin Xiao
  6. Devshree Patel
  7. Ratnam Parikh
  8. Yesha Shastri

Citations in the corpus