Multimodal Unified Attention Networks for Vision-and-Language Interactions

By Zhou Yu, Yuhao Cui, Jun Yu, Dacheng Tao, Qi Tian. 2019

48 citations recorded by the source corpus.

Learning an effective attention mechanism for multimodal data is important in many vision-and-language tasks that require a synergic understanding of both the visual and textual contents. Existing state-of-the-art approaches use co-attention models to associate each visual object (e.g., image region) with each textual object (e.g., query word). Despite the success of these co-attention models, they only model inter-modal interactions while neglecting intra-modal interactions. Here we propose a general `unified attention' model that simultaneously captures the intra- and inter-modal interactions of multimodal features and outputs their corresponding attended representations. By stacking such unified attention blocks in depth, we obtain the deep Multimodal Unified Attention Network (MUAN), which can seamlessly be applied to the visual question answering (VQA) and visual grounding tasks. We evaluate our MUAN models on two VQA datasets and three visual grounding datasets, and the results show that MUAN achieves top-level performance on both tasks without bells and whistles.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Mohit Bansal
2. Hao Hao Tan
3. Rongrong Ji
4. Xiaoshuai Sun
5. Yiyi Zhou
6. Liujuan Cao
7. Gen Luo
8. Feiyue Huang

In Computer Vision and Pattern Recognition

1. Mohit Bansal
2. Luc Van Gool
3. Dacheng Tao
4. Trevor Darrell
5. Rongrong Ji
6. Andrew Zisserman
7. Cordelia Schmid
8. Jitendra Malik

In Representation & Self-Supervised Learning

1. Mohit Bansal
2. Hao Hao Tan
3. Rongrong Ji
4. Xiaoshuai Sun
5. Shimon Ullman
6. Feiyue Huang
7. Gen Luo
8. Yiyi Zhou

Citations in the corpus

• LXMERT: Learning Cross-Modality Encoder Representations from Transformers (2019, background): “MUAN achieves 71.1% (compared to our 72.5%).”
• Towards Lightweight Transformer Via Group-Wise Transformation for Vision-and-Language Tasks (2022, background): “present, Transformer and its variants have been applied to a variety of vision-and-language tasks, such as visual question answering (VQA) [2], [6], [7], image captioning [3], [8]– [10], and referring expression comprehension [4], [11], achieving dominant performance in multiple benchmarks.”
• VQA With No Questions-Answers Training (2018, background): “Many recent works focused on improving the image-question fused features [17, 9, 84, 10, 16], attention mechanisms for selecting important features [80, 54, 47, 4, 12, 59, 50, 27], including self and guided attention [83, 21, 82], applying pre-trained networks [65, 46, 86], and incorporating outputs”