Improved Fusion of Visual and Language Representations by Dense Symmetric Co-attention for Visual Question Answering

By Duy-Kien Nguyen, Takayuki Okatani. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018

305 citations recorded by the source corpus.

A key solution to visual question answering (VQA) exists in how to fuse visual and language features extracted from an input image and question. We show that an attention mechanism that enables dense, bi-directional interactions between the two modalities contributes to boost accuracy of prediction of answers. Specifically, we present a simple architecture that is fully symmetric between visual and language representations, in which each question word attends on image regions and each image region attends on question words. It can be stacked to form a hierarchy for multi-step interactions between an image-question pair. We show through experiments that the proposed architecture achieves a new state-of-the-art on VQA and VQA 2.0 despite its small size. We also present qualitative evaluation, demonstrating how the proposed attention mechanism can generate reasonable attention maps on images and questions, which leads to the correct answer prediction.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/f7cc85bed2a3d0b0ef1c0e0258f5b60ee4bb4622

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Integrating Text and Image: Determining Multimodal Document Intent in Instagram Posts (2019, methodology): “Although naive, this strategy has been shown to be effective at different tasks such as Visual Question Answering (Nguyen and Okatani, 2018) and image-caption matching (Ahuja et al., 2018).”
Attention, please! A survey of neural attention models in deep learning (2021, unknown): “(2018a), Nguyen and Okatani (2018), Andreas et al.”
Multimodal Fusion with Co-Attention Networks for Fake News Detection (2021, background): “Co-attention block (Lu et al., 2019) is a variant of the standard multi-head self-attention (MSA) block (Vaswani et al., 2017), which can capture global dependencies of all positions in a sequence and is widely used in NLP and VQA tasks (Nguyen and Okatani, 2018; Gao et al., 2019).”
NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenario (2023, methodology): “In the image-based VQA models, earlier works [1, 30, 32, 35] typically use CNNs to extract image features, and RNNs to process the question.”
Multi-Task Learning with Deep Neural Networks: A Survey (2020, background): “(Nguyen and Okatani, 2019) introduces an architecture for shared vision and language tasks by using dense co-attention layers (Nguyen and Okatani, 2018), in which tasks are organized into a hierarchy and low-level tasks are supervised at earlier layers in the network.”
Multimodal Unified Attention Networks for Vision-and-Language Interactions (2019, methodology): “, X ↔ Y ) [19][20], our unified attention models the intra-modal and inter-modal interactions simultaneously (i.”
Multimodal Co-Attention Transformer for Survival Prediction in Gigapixel Whole Slide Images (2021, background): “…in Visual Question Answering (VQA) that learn relationships attributing how word embed-dings attend to salient objects in an image when answering a question [34, 29, 64, 28, 44, 63], in our framework, we learn how instance-level histology features attend to genes when predicting patient survival.”
Multi-View Attention Networks for Visual Dialog (2020, background): “More advanced approaches have been well studied in Nguyen and Okatani (2018); Yu et al. (2019).”
Efficient Attention Mechanism for Handling All the Interactions between Many Inputs with Application to Visual Dialog (2019, methodology): “Speciﬁcally, for VQA, it is proposed to use intra-modal and inter-modal attention blocks and stack them alternately to fuse question and image features [14]; it is also proposed to use a cascade of modular co-attention layers that compute the self-attention and guided-attention of question and image”
History for Visual Dialog: Do we really need it? (2020, methodology): “Different from previous co-attention networks (Kim et al., 2018; Nguyen and Okatani, 2018), MCA networks use guided attention to model dense relations between the question and image regions for better visual grounding.”
Modality-Balanced Models for Visual Dialogue (2020, background): “Co-attention (Lu et al. 2016) is proposed to integrate information from different modalities (i.e., image and language) and more advanced approaches have shown good performance (Yu et al. 2017a; Nam, Ha, and Kim 2017; Nguyen and Okatani 2018).”
Compact Trilinear Interaction for Visual Question Answering (2019, methodology): “There are different approaches have been proposed for VQA [18, 5, 8, 45, 20, 42, 24, 2, 28, 23, 38, 46, 29, 40].”
Mining Cross-Image Semantics for Weakly Supervised Semantic Segmentation (2020, background): “Further, co-attention mechanisms become an essential tool in many vision-language applications and sequential modeling tasks, such as visual question answering [39,42,76,79], visual dialog [74,85], vision-language navigation [66], and video segmentation [40,60], showing its effectiveness in capturin”
Reasoning with Heterogeneous Graph Alignment for Video Question Answering (2020, background): “The co-attention mechanism is viewed as another effective solution, and previous work has suggested a variety of task-speciﬁc structures (Nguyen and Okatani 2018; Gao et al. 2019; Yu et al. 2019; Li et al. 2019).”
FSA-Net: Learning Fine-Grained Structure Aggregation for Head Pose Estimation From a Single Image (2019, unknown): “Our attention can be optimized in an end-to-end manner along with the pose estimation without complex additional techniques [18, 19, 40, 30, 17].”
See More, Know More: Unsupervised Video Object Segmentation With Co-Attention Siamese Networks (2019, background): “More recently, co-attention mechanisms have been studied in vision-and-language tasks, such as visual question answering [35, 65, 63, 39] and visual dialogue [63].”