Dynamic Fusion With Intra- and Inter-Modality Attention Flow for Visual Question Answering

By Peng Gao, Zhengkai Jiang, Haoxuan You, Pan Lu, Steven C. H. Hoi, Xiaogang Wang, Hongsheng Li. 2018

416 citations recorded by the source corpus.

Learning effective fusion of multi-modality features is at the heart of visual question answering. We propose a novel method of dynamically fuse multi-modal features with intra- and inter-modality information flow, which alternatively pass dynamic information between and across the visual and language modalities. It can robustly capture the high-level interactions between language and vision domains, thus significantly improves the performance of visual question answering. We also show that, the proposed dynamic intra modality attention flow conditioned on the other modality can dynamically modulate the intra-modality attention of the current modality, which is vital for multimodality feature fusion. Experimental evaluations on the VQA 2.0 dataset show that the proposed method achieves the state-of-the-art VQA performance. Extensive ablation studies are carried out for the comprehensive analysis of the proposed method.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

LXMERT: Learning Cross-Modality Encoder Representations from Transformers (2019, background): “, 2018), DFAF (Gao et al., 2019a), and Cycle-Consistency (Shah et al.”
How Much Can CLIP Benefit Vision-and-Language Tasks? (2021, neutral): “Various methods have been introduced (Fukui et al., 2016; Yang et al., 2016; Anderson et al., 2018a; Jiang et al., 2018; Gao et al., 2019; Jiang et al., 2020).”
Attention on Attention for Image Captioning (2019, methodology): “AoA generates an attention gate and an information vector via two linear transformations and applies the gate to the vector to add a second attention, where the techniques are similar to some other work: GLU [10], which replaces RNN and CNN to capture long-range dependencies for language modeling; m”
LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention (2023, methodology): “In Table 2, we compare LLaMA-Adapter with popular VQA methods [2, 9, 17, 18, 22, 23, 28, 53] and language models [5, 16, 59].”
Multimodal Chain-of-Thought Reasoning in Language Models (2023, background): “(2022a), our baselines include (i) Visual question answering (VQA) models (Anderson et al., 2018; Kim et al., 2018; Yu et al., 2019; Gao et al., 2019; Kim et al., 2021; Lu et al., 2021; Li et al., 2019); (ii) Text-to-text LM models.”
Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering (2022, neutral): “Our experiments show that current multimodal methods [54, 1, 20, 8, 25, 34] fail to achieve satisfactory performance on S CIENCE QA and do not generate correct explanations.”
CLIP-Adapter: Better Vision-Language Models with Feature Adapters (2021, neutral): “Previously, attention-based approaches such as bottom-up top-down attention [2], BAN [30], Intra-Inter [14], and MCAN [70] had dominated visual-language tasks.”
Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling (2021, background): “As vision and language usually contain complementary information, joint learning of vision and language representations has been proven to be quite effective on multimodality tasks, such as Visual Question Answering [1, 2, 18, 33], Image Captioning [29, 67], and Referring Expression [68].”
Modality Shifting Attention Network for Multi-Modal Video Question Answering (2020, methodology): “Relationships of HAM to Other Methods Recent studies in VQA [11, 34] have shown that simultaneously learning self-attention and co-attention for visual and textual modalities leads to a more accurate prediction.”
A Comprehensive Overview of Large Language Models (2023, neutral): “Different from previous works [230], [231] that rely on limited VQA datasets and small-scale neural networks, current LLMaided methods offer benefits of stronger generalization ability, emergent ability, and interactivity [214].”
RUBi: Reducing Unimodal Biases in Visual Question Answering (2019, background): “…such as ﬁne-grained recognition, object detection, counting, activity recognition, commonsense reasoning, etc. Current end-to-end VQA models [15–22] achieve impressive results on most of these benchmarks and are Figure 1: Our RUBi approach aims at reducing the amount of unimodal biases learned…”
Towards Lightweight Transformer Via Group-Wise Transformation for Vision-and-Language Tasks (2022, background): “have been proposed to achieve the state-of-the-art performance in VQA [2], [6], [7], [48] and image captioning [3], [9], [10], [49]–[53].”
A Survey of Deep Learning for Mathematical Reasoning (2022, neutral): “In this domain, visual inputs are encoded using ResNet (He et al., 2016) or Faster-RCNN (Ren et al., 2015), while textual representations are obtained via GRU or LTSM. Subsequently, the joint representation is learned using multimodal fusion models, such as BAN (Kim et al., 2018), FiLM (Perez et al.”
Unsupervised Vision-and-Language Pre-training Without Parallel Images and Captions (2021, methodology): “Models developed before BERT are listed as Pre-BERT (Gao et al. (2019) for VQA, Suhr et al.”
TRAR: Routing the Attention Spans in Transformer for Visual Question Answering (2021, background): “The rapid development of VQA has been supported by the emergence of various benchmark datasets [1, 3, 14, 28, 29, 32, 52] and methods [13, 59, 66, 72, 54, 46, 77, 78, 79, 80].”