Deep Modular Co-Attention Networks for Visual Question Answering

By Zhou Yu, Jun Yu, Yuhao Cui, Dacheng Tao, Qi Tian. 2019

996 citations recorded by the source corpus.

Visual Question Answering (VQA) requires a fine-grained and simultaneous understanding of both the visual content of images and the textual content of questions. Therefore, designing an effective `co-attention' model to associate key words in questions with key objects in images is central to VQA performance. So far, most successful attempts at co-attention learning have been achieved by using shallow models, and deep co-attention models show little improvement over their shallow counterparts. In this paper, we propose a deep Modular Co-Attention Network (MCAN) that consists of Modular Co-Attention (MCA) layers cascaded in depth. Each MCA layer models the self-attention of questions and images, as well as the question-guided-attention of images jointly using a modular composition of two basic attention units. We quantitatively and qualitatively evaluate MCAN on the benchmark VQA-v2 dataset and conduct extensive ablation studies to explore the reasons behind MCAN's effectiveness. Experimental results demonstrate that MCAN significantly outperforms the previous state-of-the-art. Our best single model delivers 70.63% overall accuracy on the test-dev set.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Jianfeng Gao
  3. Zhe Gan
  4. Yejin Choi
  5. Percy Liang
  6. Caiming Xiong
  7. Dacheng Tao
  8. Luke S. Zettlemoyer

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Xiaogang Wang
  4. Trevor Darrell
  5. Ming-Hsuan Yang
  6. Cordelia Schmid
  7. Jitendra Malik
  8. Andrew Zisserman

In Multimodal Learning

  1. Zhe Gan
  2. Linjie Li
  3. Jingjing Liu
  4. Jianfeng Gao
  5. Lijuan Wang
  6. Mohit Bansal
  7. Devi Parikh
  8. Yu Cheng

In Question Answering & Reading Comprehension

  1. Anton van den Hengel
  2. Weizhe Lin
  3. Bill Byrne
  4. Jingjing Liu
  5. Linjie Li
  6. Yueting Zhuang
  7. Mohit Bansal
  8. Yu Cheng

Citations in the corpus