Deep Modular Co-Attention Networks for Visual Question Answering
By Zhou Yu, Jun Yu, Yuhao Cui, Dacheng Tao, Qi Tian. 2019
996 citations recorded by the source corpus.
Visual Question Answering (VQA) requires a fine-grained and simultaneous understanding of both the visual content of images and the textual content of questions. Therefore, designing an effective `co-attention' model to associate key words in questions with key objects in images is central to VQA performance. So far, most successful attempts at co-attention learning have been achieved by using shallow models, and deep co-attention models show little improvement over their shallow counterparts. In this paper, we propose a deep Modular Co-Attention Network (MCAN) that consists of Modular Co-Attention (MCA) layers cascaded in depth. Each MCA layer models the self-attention of questions and images, as well as the question-guided-attention of images jointly using a modular composition of two basic attention units. We quantitatively and qualitatively evaluate MCAN on the benchmark VQA-v2 dataset and conduct extensive ablation studies to explore the reasons behind MCAN's effectiveness. Experimental results demonstrate that MCAN significantly outperforms the previous state-of-the-art. Our best single model delivers 70.63% overall accuracy on the test-dev set.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Question Answering & Reading Comprehension
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Zhe Gan
- Yejin Choi
- Percy Liang
- Caiming Xiong
- Dacheng Tao
- Luke S. Zettlemoyer
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Xiaogang Wang
- Trevor Darrell
- Ming-Hsuan Yang
- Cordelia Schmid
- Jitendra Malik
- Andrew Zisserman
In Multimodal Learning
In Question Answering & Reading Comprehension
- Anton van den Hengel
- Weizhe Lin
- Bill Byrne
- Jingjing Liu
- Linjie Li
- Yueting Zhuang
- Mohit Bansal
- Yu Cheng
Citations in the corpus
- UNITER: Learning UNiversal Image-TExt Representations (2019, methodology): “For results on test-dev and test-std splits, both training and validation sets are used for training, and additional question-answer pairs from Visual Genome are used for data augmentation as in Yu et al. (2019).”
- VisualBERT: A Simple and Performant Baseline for Vision and Language (2019, methodology): “…of bounding region proposals as our methods (first section), our models (second section), and other incomparable methods (third section) that use external question-answer pairs from Visual Genome (+VG) , multiple detectors (Yu et al., 2019a) (+Multiple Detectors) and ensembles of their models.”
- Large-Scale Adversarial Training for Vision-and-Language Representation Learning (2020, background): “Prominent work includes: (i) advanced attention mechanisms [40, 79, 51]; (ii) better multimodal fusion methods [14, 80, 27, 26]; (iii) multi-step reasoning [77, 22, 15]; (iv) incorporation of object relations [55, 48, 6, 32]; and (v) neural module networks for compositional reasoning [3, 25, 20, 11]”
- LXMERT: Learning Cross-Modality Encoder Representations from Transformers (2019, background): “Since then, there have been some recently updated papers such as MCAN (Yu et al., 2019b), MUAN (Yu et al.”
- How Much Can CLIP Benefit Vision-and-Language Tasks? (2021, methodology): “…V&L pre-training has been proposed as an effective technique to improve the performance on various V&L tasks (Lu et al., 2019; Tan and Bansal, 2019; Li et al., 2019a; Su et al., 2019; Chen et al., 2020; Zhou et al., 2020; Huang et al., 2020; Li et al., 2020; Zhang et al., 2021; Li et al., 2021b).”
- LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention (2023, methodology): “In Table 2, we compare LLaMA-Adapter with existing popular VQA meth-ods [65, 33, 34] and large language models [27, 4, 74] on ScienceQA datset.”
- Multimodal Chain-of-Thought Reasoning in Language Models (2023, neutral): “As the two-stage method (QCMR → A) in Table 3 achieves better performance than all the one-stage method in Table 2, we choose the two-stage method in our Multimodal-CoT framework.”
- Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering (2022, neutral): “Our experiments show that current multimodal methods [54, 1, 20, 8, 25, 34] fail to achieve satisfactory performance on S CIENCE QA and do not generate correct explanations.”
- ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision (2021, methodology): “We use MCAN (Yu et al., 2019) and MaxEnt (Suhr et al.”
- Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models (2023, background): “[59] Zhou Yu, Jun Yu, Yuhao Cui, Dacheng Tao, and Qi Tian.”
- Mining Cross-Image Semantics for Weakly Supervised Semantic Segmentation (2020, background): “Further, co-attention mechanisms become an essential tool in many vision-language applications and sequential modeling tasks, such as visual question answering [39,42,76,79], visual dialog [74,85], vision-language navigation [66], and video segmentation [40,60], showing its effectiveness in capturin”
- Efficient Attention Mechanism for Handling All the Interactions between Many Inputs with Application to Visual Dialog (2019, methodology): “They follow the original idea of the Transformer, considering attention from source features Y to target features X as In MCAN [44], language feature is treated as the source and visual feature is as the target.”
- ScanQA: 3D Question Answering for Spatial Scene Understanding (2021, methodology): “Inspired by the architecture of the deep modular co-attention networks of MCAN [49], often used for VQA, we use transformer blocks [42] to represent the relationships between object proposals and between question words.”
- TRAR: Routing the Attention Spans in Transformer for Visual Question Answering (2021, methodology): “More and more researchers [39, 71, 79, 58, 22] follow the design paradigm of Transformer to propose various multi-modal networks, achieving new state-of-the-art performance on various benchmarks [14, 30, 28, 7].”
- A Comprehensive Overview of Large Language Models (2023, positive): “Likewise, PCW chunks larger inputs into the pre-trained context lengths and applies the same positional encodings to each chunk.”