Mucko: Multi-Layer Cross-Modal Knowledge Reasoning for Fact-based Visual Question Answering

By Zihao Zhu, Jiangtao Yu, Yujing Wang, Yajing Sun, Yue Hu, Qi Wu. 2020

160 citations recorded by the source corpus.

Fact-based Visual Question Answering (FVQA) requires external knowledge beyond the visible content to answer questions about an image. This ability is challenging but indispensable to achieve general VQA. One limitation of existing FVQA solutions is that they jointly embed all kinds of information without fine-grained selection, which introduces unexpected noises for reasoning the final answer. How to capture the question-oriented and information-complementary evidence remains a key challenge to solve the problem. In this paper, we depict an image by a multi-modal heterogeneous graph, which contains multiple layers of information corresponding to the visual, semantic and factual features. On top of the multi-layer graph representations, we propose a modality-aware heterogeneous graph convolutional network to capture evidence from different layers that is most relevant to the given question. Specifically, the intra-modal graph convolution selects evidence from each modality and cross-modal graph convolution aggregates relevant information across different graph layers. By stacking this process multiple times, our model performs iterative reasoning across three modalities and predicts the optimal answer by analyzing all question-oriented evidence. We achieve a new state-of-the-art performance on the FVQA task and demonstrate the effectiveness and interpretability of our model with extensive experiments.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Zhe Gan
2. Lijuan Wang
3. Zicheng Liu
4. Jianfeng Gao
5. Jianfeng Wang
6. Zhengyuan Yang
7. Xiaowei Hu
8. Bill Byrne

In Computer Vision and Pattern Recognition

1. Zhe Gan
2. Zicheng Liu
3. Lijuan Wang
4. Jianfeng Gao
5. Luc Van Gool
6. Cordelia Schmid
7. Lu Yuan
8. Dacheng Tao

In Multimodal Learning

1. Zhe Gan
2. Lijuan Wang
3. Jianfeng Wang
4. Zicheng Liu
5. Zhengyuan Yang
6. Xiaowei Hu
7. Yumao Lu
8. Bill Byrne

In Question Answering & Reading Comprehension

1. Weizhe Lin
2. Bill Byrne
3. Chitta Baral
4. Zhengyuan Yang
5. Lijuan Wang
6. Zicheng Liu
7. Zhe Gan
8. Xiaowei Hu

Citations in the corpus

• An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA (2021, methodology): “After external knowledge retrieval, studies have focused on reasoning over the knowledge acquired and the input image-question pair for answer prediction, where graph convolution network has been shown to be an effective way for multimodal fusion (Narasimhan, Lazebnik, and Schwing 2018; Zhu et al. 2”
• See, Think, Confirm: Interactive Prompting Between Vision and Language Models for Knowledge-based Visual Reasoning (2023, unknown): “Most early methods [20,23,29,40,58,71,76] use deep neural networks to understand images and retrieve relevant knowledge from explicit knowledge bases.”
• Multi-Modal Answer Validation for Knowledge-Based VQA (2021, methodology): “Most recent knowledge-based VQA systems [25, 36, 42, 24] follow a two-stage framework, where a retriever first looks up knowledge relevant to the question and the image, and then a separate comprehension model predicts the answer.”
• Retrieval Augmented Visual Question Answering with Outside Knowledge (2022, background): “(3) Mucko (Zhu et al., 2020): the current state-of-the-art system that uses GCNs to combine visual, fact, and semantic graphs.”
• Weakly-Supervised Visual-Retriever-Reader for Knowledge-based Question Answering (2021, background): “Mucko (Zhu et al., 2020) goes a step further, reasoning on visual, fact, and semantic graphs separately, and uses cross-modal networks to aggregate them together.”
• REVIVE: Regional Visual Representation Matters in Knowledge-Based Visual Question Answering (2022, background): “Recent studies [29, 28, 20, 19, 35, 31, 17, 32, 5] integrate different knowledge from various external knowledge resources, e.”