Dual Attention Networks for Multimodal Reasoning and Matching

By Hyeonseob Nam, Jung-Woo Ha, Jeonghee Kim. 2016

716 citations recorded by the source corpus.

We propose Dual Attention Networks (DANs) which jointly leverage visual and textual attention mechanisms to capture fine-grained interplay between vision and language. DANs attend to specific regions in images and words in text through multiple steps and gather essential information from both modalities. Based on this framework, we introduce two types of DANs for multimodal reasoning and matching, respectively. The reasoning model allows visual and textual attentions to steer each other during collaborative inference, which is useful for tasks such as Visual Question Answering (VQA). In addition, the matching model exploits the two attention mechanisms to estimate the similarity between images and sentences by focusing on their shared semantics. Our extensive experiments validate the effectiveness of DANs in combining vision and language, achieving the state-of-the-art performance on public benchmarks for VQA and image-text matching.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Webly Supervised Joint Embedding for Cross-Modal Image-Text Retrieval (2018, methodology): “Multi-modal attention mechanism is proposed in [40] to selectively attend to specific image regions and sentence fragments and calculate similarity.”
Combined Scaling for Zero-shot Transfer Learning (2021, neutral): “…Socher et al., 2013; Hodosh et al., 2013; Frome et al., 2013; Norouzi et al., 2013; Kiros et al., 2014; Socher et al., 2014; Akata et al., 2015b,a; Nam et al., 2017; Faghri et al., 2017; Li et al., 2019; Liu et al., 2019; Lu et al., 2019; Messina et al., 2020; Chen et al., 2020d; Huang et al.,…”
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision (2021, background): “, leveraging object detectors, dense feature maps, or multi-attention layers) (Socher et al., 2014; Karpathy et al., 2014; Kiros et al.; Nam et al., 2017; Li et al., 2019; Messina et al., 2020; Chen et al., 2020a) have been proposed.”
Bilinear Attention Networks (2018, background): “The attention mechanism of co-attention networks [36, 18, 20, 39] concurrently infers visual and textual attention distributions for each modality.”
Stacked Cross Attention for Image-Text Matching (2018, background): “[31] proposed Dual Attentional Network to capture fine-grained interplay between vision and language through multiple steps.”
Deep Modular Co-Attention Networks for Visual Question Answering (2019, background): “proposed a multi-stage coattention learning model to refine the attentions based on memory of previous attentions [23].”
BAM: Bottleneck Attention Module (2018, methodology): “Another way of doing this is to use bi-directional inferring, producing attention maps for both text and images, as suggested in [36].”
Beyond Bilinear: Generalized Multimodal Factorized High-Order Pooling for Visual Question Answering (2017, background): “egories: 1) the coarse joint embedding models [6], [9], [14]; 2) the fine-grained joint embedding models with attention [8], [10], [15]–[17]; and 3) the external knowledge-based models [18], [19].”
Learning Joint Embedding with Multimodal Cues for Cross-Modal Video-Text Retrieval (2018, methodology): “Most recent works relating to text and imagemodality are trained with ranking loss [7, 9, 18, 23, 34].”
Multi-step Reasoning via Recurrent Dual Attention for Visual Dialog (2019, background): “The idea of multi-step reasoning has been explored in many tasks, including image classiﬁ-cation [34], text classiﬁcation [54], image generation [17], language-based image editing [6], Visual Question Answering (VQA) [53, 36], and Machine Reading Comprehension (MRC) [8, 12, 20, 44, 41, 31].”
Multi-modal Factorized Bilinear Pooling with Co-attention Learning for Visual Question Answering (2017, methodology): “Nam et al. proposed a multi-stage co-attention learning framework to reﬁne the attentions based on memory of previous attentions [21].”
VSE++: Improved Visual-Semantic Embeddings (2017, methodology): “Works such as Karpathy & Fei-Fei (2015), Socher et al. (2014), Vendrov et al. (2015), Wang et al. (2017), Huang et al. (2016), Nam et al. (2016) use a similar loss to optimize more sophisticated models.”
Interactive Speech and Noise Modeling for Speech Enhancement (2020, methodology): “Two-branch neural networks have been explored in various tasks for capturing cross-modality information (Nam, Ha, and Kim 2017; Wang et al. 2019) or different levels of information (Simonyan and Zisserman 2014; Wang et al. 2020).”
Image-Question-Answer Synergistic Network for Visual Dialog (2019, methodology): “This starts with the linear combination of two features, such as the stacked attention network [30] by learning the attention using multi-step reasoning and the dual attention network [20] that learns both the visual and textual attention.”