Relation-Aware Graph Attention Network for Visual Question Answering

By Linjie Li, Zhe Gan, Yu Cheng, Jingjing Liu. 2019

394 citations recorded by the source corpus.

In order to answer semantically-complicated questions about an image, a Visual Question Answering (VQA) model needs to fully understand the visual scene in the image, especially the interactive dynamics between different objects. We propose a Relation-aware Graph Attention Network (ReGAT), which encodes each image into a graph and models multi-type inter-object relations via a graph attention mechanism, to learn question-adaptive relation representations. Two types of visual object relations are explored: (i) Explicit Relations that represent geometric positions and semantic interactions between objects; and (ii) Implicit Relations that capture the hidden dynamics between image regions. Experiments demonstrate that ReGAT outperforms prior state-of-the-art approaches on both VQA 2.0 and VQA-CP v2 datasets. We further show that ReGAT is compatible to existing VQA architectures, and can be used as a generic relation encoder to boost the model performance for VQA.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

VisualBERT: A Simple and Performant Baseline for Vision and Language (2019, background): “Li et al. (2019) take a further step and explicitly build a graph to encode object relations.”
Large-Scale Adversarial Training for Vision-and-Language Representation Learning (2020, background): “Prominent work includes: (i) advanced attention mechanisms [40, 79, 51]; (ii) better multimodal fusion methods [14, 80, 27, 26]; (iii) multi-step reasoning [77, 22, 15]; (iv) incorporation of object relations [55, 48, 6, 32]; and (v) neural module networks for compositional reasoning [3, 25, 20, 11]”
Scaling Local Self-Attention for Parameter Efficient Visual Backbones (2021, background): “…scaling of convolution; (c) empirical ability to capture long-range dependencies for use in larger images; (d) ﬂexibility to handle and integrate multiple types of data that appear in vision, such as pixels [61, 3, 46, 68], point clouds [65], sequence conditioning information [64], and graphs [32].”
Multi-step Reasoning via Recurrent Dual Attention for Visual Dialog (2019, methodology): “Finally, we have tried using different image feature inputs, and incorporating relation-aware encoders (Li et al., 2019) into ReDAN to further boost the performance.”
Learning by Abstraction: The Neural State Machine (2019, methodology): “24.96 HAN [59] 28.65 GVQA [3] 31.30 RAMEN [73] 39.21 BAN [46] 39.31 MuRel [15] 39.54 ReGAT [52] 40.42 NSM 45.80 First, we measure the performance on VQA-CP [3], which provides a new split of the VQA2 dataset [27], where the answer distribution is kept different between the training and the test…”
An Empirical Study on Leveraging Scene Graphs for Visual Question Answering (2019, background): “One direction is to leverage object relationships for better scene understanding [32, 43, 46, 58, 59].”
Meta Module Network for Compositional Visual Reasoning (2019, background): “Monolithic Networks : Most monolithic networks for visual reasoning resort to attention mechanism for multimodal fusion (Zhu et al., 2017; 2016; Zhou et al., 2017; Yu et al., 2019; 2017; Kim et al., 2016; 2018; Kaﬂe et al., 2018; Li et al., 2019; Hu et al., 2019).”
ConceptBert: Concept-Aware Representation for Visual Question Answering (2020, background): “XNM Net (Shi et al., 2018) and ReGAT (Li et al., 2019) are designed to answer semantically-complicated questions.”
Deconfounded Image Captioning: A Causal Retrospect (2020, background): “Such a strategy is also widely applied in other vision-language tasks like VQA [50], [51], Grounding [52], [53], and Visual Dialogue [54], e.g., [54] exploits the scene graph containing both visual and semantic knowledge of the image to complete the dialogue.”
ROSITA: Enhancing Vision-and-Language Semantic Alignments via Cross- and Intra-modal Knowledge Integration (2021, methodology): “Inspired by [14, 20, 43], we calculate the similarity between each paired regions by their Intersection over Union (IoU) score.”
Bilinear Graph Networks for Visual Question Answering (2019, background): “Relation-aware Graph Attention Network (ReGAT) [39] involves the explicit relationship between objects, including spatial and semantic graphs.”
DualVD: An Adaptive Dual Encoding Model for Deep Visual Understanding in Visual Dialogue (2019, neutral): “…discriminative models, namely LF (Das et al. 2017), HRE (Das et al. 2017), MN (Das et al. 2017), SAN-QI (Yang et al. 2016), HieCoAtt-QI (Lu et al. 2016), AMEM (Seo et al. 2017), HCIAE (Lu et al. 2017), SF (Jain, Lazeb-nik, and Schwing 2018), CoAtt (Qi et al. 2018), CorefMN (Kottur et…”
Learning to Ground Visual Objects for Visual Dialog (2021, background): “…et al., 2016, 2018; Cornia et al., 2020; Ghanimifard and Dobnik, 2019), visual question answering (Ren et al., 2015a; Gao et al., 2015; Lu et al., 2016; Anderson et al., 2018; Li et al., 2019; Huang et al., 2020) and visual dialog (Das et al., 2017; Chen et al., 2021a; ∗Corresponding author.”
Coarse-to-Fine Reasoning for Visual Question Answering (2021, methodology): “…into three groups: joint learning mechanisms: BAN [26], Pythia [24], DFAF [15], fPMC [20], STL [54], CTI [11], and MCAN [59]; reasoning-based methods: Murel [4], Re-GAT [30], MMN [6], NMS [22], and HAN [25]; and large-scale visual-language modeling: LXMERT [47], OSCAR [32], and UNITER [7].”
Recent Advances and Trends in Multimodal Deep Learning: A Review (2021, background): “[78] proposed a relation aware graph attention mechanism for VQA.”