Boosting Visual Question Answering with Context-aware Knowledge Aggregation

By Guohao Li, Xin Wang, Wenwu Zhu. 2020

102 citations recorded by the source corpus.

Given an image and a natural language question, Visual Question Answering (VQA) aims at answering the textual question correctly. Most VQA approaches in literature targets at finding answers to the questions solely based on analyzing the given images and questions alone. Other works that try to incorporate external knowledge into VQA adopt a query-based search on knowledge graphs to obtain the answer. However, these works suffer from the following problem: the model training process heavily relies on the ground-truth knowledge facts which serve as supervised information --- missing these ground-truth knowledge facts during training will lead to failures in producing the correct answers. To solve the challenging issue, we propose a Knowledge Graph Augmented (KG-Aug) model which conducts context-aware knowledge aggregation on external knowledge graphs, requiring no ground-truth knowledge facts for extra supervision. The proposed KG-Aug model is capable of retrieving context-aware knowledge subgraphs given visual images and textual questions, and learning to aggregate the useful image- and question-dependent knowledge which is then utilized to boost the accuracy in answering visual questions. We carry out extensive experiments to validate the effectiveness of our proposed KG-Aug models against several baseline approaches on various datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jianfeng Gao
2. Zhe Gan
3. Mohit Bansal
4. Lijuan Wang
5. Zicheng Liu
6. Jianfeng Wang
7. Trevor Darrell
8. Yejin Choi

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Trevor Darrell
3. Jianfeng Gao
4. Cordelia Schmid
5. Zhe Gan
6. Dacheng Tao
7. Jitendra Malik
8. Abhinav Kumar Gupta

In Question Answering & Reading Comprehension

1. Weizhe Lin
2. Bill Byrne
3. Lijuan Wang
4. Jianfeng Gao
5. Xiaowei Hu
6. Zhengyuan Yang
7. Zicheng Liu
8. Jiasen Lu

Citations in the corpus

• KAT: A Knowledge Augmented Transformer for Vision-and-Language (2021, methodology): “…et al., 2019) - 20.67 BAN (Marino et al., 2019) - 25.1 BAN+AN (Marino et al., 2019) Wikipedia 25.61 MUTAN (Marino et al., 2019) - 26.41 BAN+KG-AUG (Li et al., 2020b) Wikipedia+ConceptNet 26.71 MUTAN+AN (Marino et al., 2019) Wikipedia 27.84 ConceptBERT (Garderes et al., 2020) ConceptNet 33.66…”
• An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA (2021, methodology): “Instead of treating VQA as a classification task over a pre-selected answer vocabulary (Goyal et al. 2017; Li et al. 2020), we predict the answer in an open-ended text generation manner.”
• Multi-Modal Answer Validation for Knowledge-Based VQA (2021, methodology): “More recent VQA systems (Lu et al. 2019; Tan and Bansal 2019; Liu et al. 2019; Li et al. 2019; Yu et al. 2019; Li et al. 2020; Zhou et al. 2020; Chen et al. 2020; Lu et al. 2020) first extract visual features from a pre-trained object detector.”
• REVIVE: Regional Visual Representation Matters in Knowledge-Based Visual Question Answering (2022, unknown): “4 BAN+KG-AUG [15] Wikipedia+ConceptNet 26.”
• Retrieval Augmented Visual Question Answering with Outside Knowledge (2022, background): “KB-VQA can access both structured data, such as ConceptNet and other KGs (Narasimhan et al., 2018a; Garderes et al., 2020; Li et al., 2020a; Wu et al., 2022; Marino et al., 2021), as well as unstructured data such as Wikipedia passages (Wu et al., 2022; Gao et al., 2022; Gui et al., 2021).”
• KRISP: Integrating Implicit and Symbolic Knowledge for Open-Domain Knowledge-Based VQA (2020, methodology): “For improved generalization to open-domain VQA, [26, 52, 40, 80] embed the symbolic knowledge to an implicit embedding loosing the semantics of the symbols, but therefore are able to easily integrate the embedding with standard VQA approaches.”