Heterogeneous Memory Enhanced Multimodal Attention Model for Video Question Answering

By Chenyou Fan, Xiaofan Zhang, Shu Zhang, Wensheng Wang, Chi Zhang, Heng Huang. 2019

319 citations recorded by the source corpus.

In this paper, we propose a novel end-to-end trainable Video Question Answering (VideoQA) framework with three major components: 1) a new heterogeneous memory which can effectively learn global context information from appearance and motion features; 2) a redesigned question memory which helps understand the complex semantics of question and highlights queried subjects; and 3) a new multimodal fusion layer which performs multi-step reasoning by attending to relevant visual and textual hints with self-updated attention. Our VideoQA model firstly generates the global context-aware visual and textual features respectively by interacting current inputs with memory contents. After that, it makes the attentional fusion of the multimodal visual and textual representations to infer the correct answer. Multiple cycles of reasoning can be made to iteratively refine attention weights of the multimodal data and improve the final representation of the QA pair. Experimental results demonstrate our approach achieves state-of-the-art performance on four VideoQA benchmark datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Cordelia Schmid
  2. Mohit Bansal
  3. Trevor Darrell
  4. Steven C. H. Hoi
  5. Yoshua Bengio
  6. Luc Van Gool
  7. Chuang Gan
  8. Tat-seng Chua

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Ming-Hsuan Yang
  4. Cordelia Schmid
  5. Trevor Darrell
  6. Jitendra Malik
  7. Xiaogang Wang
  8. Andrew Zisserman

In Multimodal Learning

  1. Cordelia Schmid
  2. Chuang Gan
  3. Josef Sivic
  4. Josh Tenenbaum
  5. Ivan Laptev
  6. Antoine Miech
  7. Antonio Torralba
  8. Zhenfang Chen

In Question Answering & Reading Comprehension

  1. Tat-seng Chua
  2. Junbin Xiao
  3. Angela Yao
  4. Xindi Shang
  5. Jun Liu
  6. Li Xu
  7. Svetha Venkatesh
  8. Vuong Le

Citations in the corpus