Straight to the Facts: Learning Knowledge Base Retrieval for Factual Visual Question Answering

By Medhini Narasimhan, Alexander Gerhard Schwing. 2018

119 citations recorded by the source corpus.

Question answering is an important task for autonomous agents and virtual assistants alike and was shown to support the disabled in efficiently navigating an overwhelming environment. Many existing methods focus on observation-based questions, ignoring our ability to seamlessly combine observed content with general knowledge. To understand interactions with a knowledge base, a dataset has been introduced recently and keyword matching techniques were shown to yield compelling results despite being vulnerable to misconceptions due to synonyms and homographs. To address this issue, we develop a learning-based approach which goes straight to the facts via a learned embedding space. We demonstrate state-of-the-art results on the challenging recently introduced fact-based visual question answering dataset, outperforming competing methods by more than \(5\%\).

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Question Answering & Reading Comprehension

Citations in the corpus

An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA (2021, background): “Previous studies (Wang et al. 2015; Narasimhan and Schwing 2018; Wang et al. 2017; Narasimhan, Lazebnik, and Schwing 2018; Zhu et al. 2020; Li, Wang, and Zhu 2020; Marino et al. 2021; Wu et al. 2021) proposed various ways of retrieving and using the knowledge, and considered it necessary to use mult”
OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge (2019, background): “Given the question, image, and Wikipedia sentences, there is now enough information to answer the question: Teddy Roosevelt! More recent research has started to look at how to incorporate knowledge-based methods into VQA [29, 30, 36, 37].”
Image-Question-Answer Synergistic Network for Visual Dialog (2019, background): “[21] queries the triplet (visual concept, relation, attribute) in Fvqa [26] to score the retrieved facts.”
Mucko: Multi-Layer Cross-Modal Knowledge Reasoning for Fact-based Visual Question Answering (2020, methodology): “[ Narasimhan et al. , 2018 ] applies GCN on the fact graph where each node is represented by the ﬁxed form of image-question-entity embedding.”
BERT Representations for Video Question Answering (2020, background): “Besides these improvements, many other related work, such as [33, 25, 30, 12, 3, 46, 11, 1, 16, 2, 34], has contributed to the advancement of the ﬁeld.”
KRISP: Integrating Implicit and Symbolic Knowledge for Open-Domain Knowledge-Based VQA (2020, background): “Symbolic knowledge from knowledge bases is commonly represented as graphs/knowledge bases [39, 54, 55, 73, 74] or textual knowledge sources such as Wikipedia [51, 77].”
Recent Advances and Trends in Multimodal Deep Learning: A Review (2021, background): “Schwing [106] proposed a framework for the VQA task using external knowledge resources that contain a set of facts.”
Multi-Modal Answer Validation for Knowledge-Based VQA (2021, methodology): “Narasimhan and Schwing (2018) propose to retrieve relevant facts from a knowledge base.”
Boosting Visual Question Answering with Context-aware Knowledge Aggregation (2020, methodology): “Different from previous methods, several works [17, 18, 26] formulate this problem as a retrieval task that localizes the most relevant facts (i.e., subject-relation-object triplets) from the given knowledge base, where a learnable score function is usually applied to model the relevance between…”
REVIVE: Regional Visual Representation Matters in Knowledge-Based Visual Question Answering (2022, background): “Recent studies [29, 28, 20, 19, 35, 31, 17, 32, 5] integrate different knowledge from various external knowledge resources, e.”
ConceptBert: Concept-Aware Representation for Visual Question Answering (2020, methodology): “Recent research has started integrating knowledge-based methods into VQA models (Wang et al., 2017, 2016; Narasimhan et al., 2018; Narasimhan and Schwing, 2018; Zhu et al., 2015; Marino et al., 2019).”