Building a Large-scale Multimodal Knowledge Base for Visual Question Answering

By Yuke Zhu, Ce Zhang, Christopher R'e, Li Fei-Fei. 2015

34 citations recorded by the source corpus.

The complexity of the visual world creates significant challenges for comprehensive visual understanding. In spite of recent successes in visual recognition, a lack of common sense knowledge and the insufficiencies of joint reasoning still leave a large gap between computer vision models and their human counterparts for higher-level tasks such as question answering (QA). In this work, we build a multimodal knowledge base (KB) incorporating visual, textual and structured data, as well as their diverse relations for visual QA. Such a general-purpose approach presents major challenges in terms of scalability and quality. Our approach is to cast large-scale MRFs into a KB representation, and introduce a scalable knowledge base construction system by leveraging database techniques. We unify several visual QA tasks in a principled query language. Our KB achieves competitive results compared to purpose-built models, while exhibiting greater flexibility in visual QA.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Question Answering & Reading Comprehension

Citations in the corpus

Ask Me Anything: Free-Form Visual Question Answering Based on Knowledge from External Sources (2015, background): “l natural language Question Answering (QA) systems [3,10]. However, VQA systems exploiting KBs are still relatively rare. Gao et al. [11] and Malinowski et al. [20] do not use a KB at all. Zhu et al. [33] do use a KB, but it is created speciﬁcally for the purpose, and consequently contains a small a”
Explicit Knowledge-based Reasoning for Visual Question Answering (2015, background): “[43] as they use a KB and RDBMS to answer image-based questions.”
FVQA: Fact-Based Visual Question Answering (2016, methodology): “[49] used a KB and RDBMS to answer image-based queries.”
What Value Do Explicit High Level Concepts Have in Vision to Language Problems? (2015, background): “Zhu et al. [59] cast a MRF model into a Knowledge Base representation to answer commonsense-related visual questions.”
Neural Motifs: Scene Graph Parsing with Global Context (2017, background): “Deep sequential models have demonstrated strong performance for tasks such as captioning [6, 9, 40, 18] and visual question answer [1, 34, 48, 12, 8], including for problems not classically thought of as sequence problems, such as language parsing [39], or multi-label classification [41].”
KRISP: Integrating Implicit and Symbolic Knowledge for Open-Domain Knowledge-Based VQA (2020, background): “There have been many knowledge bases proposed for knowledge-based reasoning, both language-only and multi-modal [85, 14, 17, 65, 88, 87, 10, 53, 36].”
Situation Recognition: Visual Semantic Role Labeling for Image Understanding (2016, background): “At least partially motivated by the same concerns, there are recent efforts to formulate Visual Question Answering (VQA) tasks [1, 39, 51, 18, 9], where the system must answer questions like “What is the person using to cut the grass?” In a pilot study on a VQA dataset [1], we found that up to 20% o”
Leveraging Visual Question Answering for Image-Caption Ranking (2016, background): “[43] uses a knowledge base learned on object categories, attributes, actions and object affordances for query-based image retrieval.”
ConceptBert: Concept-Aware Representation for Visual Question Answering (2020, methodology): “Recent research has started integrating knowledge-based methods into VQA models (Wang et al., 2017, 2016; Narasimhan et al., 2018; Narasimhan and Schwing, 2018; Zhu et al., 2015; Marino et al., 2019).”
Straight to the Facts: Learning Knowledge Base Retrieval for Factual Visual Question Answering (2018, unknown): “[55], Wu et al .”
A Survey on Food Computing (2018, unknown): “Lu et al. 2016; Zhu and Jiang2018] and generation of the scene graph [Johnson et al. 2015] from images. These technologies are helpful for constructing the visual web [Jain2015]. Other works such as [Zhu et al. 2015] tried to build a large-scale multimodal knowledge base system to support visual que”
Image Captioning and Visual Question Answering Based on Attributes and External Knowledge (2016, unknown): “[58] used a hand-crafted KB primarily contain-”
Image Captioning and Visual Question Answering Based on Attributes and Their Related External Knowledge (2016, methodology): “Zhu et al. [67], in contrast used a hand-crafted KB primarily containing image-related information such as category labels, attribute labels and affordance labels, but also some quantities relating to their speciﬁc question format such as GPS coordinates and similar.”