KRISP: Integrating Implicit and Symbolic Knowledge for Open-Domain Knowledge-Based VQA

By Kenneth Marino, Xinlei Chen, Devi Parikh, Abhinav Kumar Gupta, Marcus Rohrbach. 2020

247 citations recorded by the source corpus.

One of the most challenging question types in VQA is when answering the question requires outside knowledge not present in the image. In this work we study open-domain knowledge, the setting when the knowledge required to answer a question is not given/annotated, neither at training nor test time. We tap into two types of knowledge representations and reasoning. First, implicit knowledge which can be learned effectively from unsupervised language pretraining and supervised training data with transformer-based models. Second, explicit, symbolic knowledge encoded in knowledge bases. Our approach combines both—exploiting the powerful implicit reasoning of transformer models for answer prediction, and integrating symbolic representations from a knowledge graph, while never losing their explicit semantics to an implicit embedding. We combine diverse sources of knowledge to cover the wide variety of knowledge needed to solve knowledge-based questions. We show our approach, KRISP (Knowledge Reasoning with Implicit and Symbolic rePresentations), significantly out-performs state-of-the-art on OK-VQA, the largest available dataset for open-domain knowledge-based VQA. We show with extensive ablations that while our model successfully exploits implicit knowledge reasoning, the symbolic answer module which explicitly connects the knowledge graph to the answer vocabulary is critical to the performance of our method and generalizes to rare answers.1

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Question Answering & Reading Comprehension, Knowledge Graphs & Representation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Question Answering & Reading Comprehension

In Knowledge Graphs & Representation

Citations in the corpus

An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA (2021, methodology): “Previous studies (Wang et al. 2015; Narasimhan and Schwing 2018; Wang et al. 2017; Narasimhan, Lazebnik, and Schwing 2018; Zhu et al. 2020; Li, Wang, and Zhu 2020; Marino et al. 2021; Wu et al. 2021) proposed various ways of retrieving and using the knowledge, and considered it necessary to use mult”
Language Models are General-Purpose Interfaces (2022, methodology): “Previous methods (Marino et al., 2021; Wu et al., 2022) typically leverage a knowledge base to ﬁlter candidate answers.”
A Survey of Knowledge Graph Reasoning on Graph Types: Static, Dynamic, and Multi-Modal (2022, background): “More recently, researchers have tended to first construct the scene KGs for a better understanding of the visual semantics, such as GRUC [186], Mucko [187], and Krisp [188].”
Multi-Modal Knowledge Graph Construction and Application: A Survey (2022, methodology): “Besides, MMKGs have advantages on providing long-tail knowledge, background knowledge, and ﬁne-grained knowledge compared with VL-PTMs [51].”
COFAR: Commonsense and Factual Reasoning in Image Search (2022, methodology): “We model implicit knowledge using MM-BERT, similar to KRISP, and for unstructured text, we use BERT embedding of the knowledge text.”
Weakly-Supervised Visual-Retriever-Reader for Knowledge-based Question Answering (2021, methodology): “Interestingly, the LXMERT baseline without utilizing any knowledge achieves better performance than KRISP (Marino et al., 2020) and ConceptBert (Gardères et al.”
Multi-Modal Answer Validation for Knowledge-Based VQA (2021, methodology): “The ﬁrst approach is parsing the knowledge in a symbolic format (Zhu et al. 2020; Narasimhan and Schwing 2018; Narasimhan, Lazebnik, and Schwing 2018; Li, Wang, and Zhu 2020; Marino et al. 2021) that usually consists of a collection of (subject, relation, object) triplets.”
PromptCap: Prompt-Guided Task-Aware Image Captioning (2022, background): “7 - KRISP [39] Feature Wikipedia + ConceptNet 38.”
Multimodal Learning With Transformers: A Survey (2022, background): “Scope This survey will discuss the multimodality specific designs of Transformer architecture including, but not limited to, the following modalities: RGB image [5], depth image [13], video [7], audio/speech/music [13], [14], [15], table [16], scene graph/layout [17], [18], [19], pose skeleton [20],”
Transform-Retrieve-Generate: Natural Language-Centric Outside-Knowledge Visual Question Answering (2022, methodology): “Other work uses vision-language models as a knowledge-free VQA model first and later adjusts the predicted answers by fusion with knowledge graphs [39] or answer validation with knowledge text [61].”
A-OKVQA: A Benchmark for Visual Question Answering using World Knowledge (2022, methodology): “KRISP is a high-performing model on OK-VQA [36].”
K-LITE: Learning Transferable Visual Models with External Knowledge (2022, background): “To address the problem, acquiring external knowledge becomes an essential component for success [93, 60, 96].”
Learning Customized Visual Models with Retrieval-Augmented Knowledge (2023, methodology): “Motivated by retrieval-augmented models in NLP, several recent works leverage visual and / or textual knowledge to improve classification [32], question answering [7, 35, 56, 60], image generation [2, 8, 49, 72], and multi-modal tasks simultaneously [62].”
Retrieval Augmented Visual Question Answering with Outside Knowledge (2022, methodology): “2: ConceptBERT , KRISP , MAVEx , and VRR .”