Incorporating External Knowledge to Answer Open-Domain Visual Questions with Dynamic Memory Networks
By Guohao Li, Hang Su, Wenwu Zhu. 2017
50 citations recorded by the source corpus.
Visual Question Answering (VQA) has attracted much attention since it offers insight into the relationships between the multi-modal analysis of images and natural language. Most of the current algorithms are incapable of answering open-domain questions that require to perform reasoning beyond the image contents. To address this issue, we propose a novel framework which endows the model capabilities in answering more complex questions by leveraging massive external knowledge with dynamic memory networks. Specifically, the questions along with the corresponding images trigger a process to retrieve the relevant information in external knowledge bases, which are embedded into a continuous vector space by preserving the entity-relation structures. Afterwards, we employ dynamic memory networks to attend to the large body of facts in the knowledge graph and images, and then perform reasoning over these facts to generate corresponding answers. Extensive experiments demonstrate that our model not only achieves the state-of-the-art performance in the visual question answering task, but can also answer open-domain questions effectively by leveraging the external knowledge.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Question Answering & Reading Comprehension
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Roozbeh Mottaghi
- Ali Farhadi
- Mohit Bansal
- Jianfeng Gao
- Jianfei Cai
- Nan Duan
- Mohammad Rastegari
- Qi Wu
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Jianfei Cai
- Xiaogang Wang
- Trevor Darrell
- Jitendra Malik
- Ming-Hsuan Yang
- Abhinav Kumar Gupta
In Multimodal Learning
- Roozbeh Mottaghi
- Ali Farhadi
- Kenneth Marino
- Jianfei Cai
- Jiuxiang Gu
- Mohammad Rastegari
- Nan Duan
- Qi Wu
In Question Answering & Reading Comprehension
Citations in the corpus
- OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge (2019, background): “The problem of using external knowledge for answering questions has been tackled by [45, 43, 44, 29, 36, 35].”
- R-VQA: Learning Visual Relation Facts with Semantic Attention for Visual Question Answering (2018, methodology): “To be specific, some methods extract highlevel semantic information [15, 31, 32], such as entities, attributes, or even retrieval results in knowledge base [15], such as DBpedia [2] and Freebase [5].”
- Mucko: Multi-Layer Cross-Modal Knowledge Reasoning for Fact-based Visual Question Answering (2020, methodology): “9 - KDMN-Ensemble [Li et al. , 2017] 60 .”
- Scene Graph Generation With External Knowledge and Image Reconstruction (2019, background): “[24] distill the retrieved context-relevant external knowledge triplet through a DMN for open-domain visual question answering.”