Explicit Knowledge-based Reasoning for Visual Question Answering

By Peng Wang, Qi Wu, Chunhua Shen, Anthony R. Dick, Anton van den Hengel. 2015

291 citations recorded by the source corpus.

We describe a method for visual question answering which is capable of reasoning about contents of an image on the basis of information extracted from a large-scale knowledge base. The method not only answers natural language questions using concepts not contained in the image, but can provide an explanation of the reasoning by which it developed its answer. The method is capable of answering far more complex questions than the predominant long short-term memory-based approach, and outperforms it significantly in the testing. We also provide a dataset and a protocol by which to evaluate such methods, thus addressing one of the key issues in general visual ques- tion answering.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Generative Adversarial Text to Image Synthesis (2016, background): “This approach was extended to incorporate an explicit knowledge base (Wang et al., 2015).”
An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA (2021, background): “Early explorations include KB-VQA (Wang et al. 2015) and F-VQA (Wang et al.”
OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge (2019, methodology): “KB-VQA [36] 2,402 700 visual reasoning with given KB Open 2.”
VizWiz Grand Challenge: Answering Visual Questions from Blind People (2018, methodology): “Over the past three years, a plethora of VQA datasets have been publicly shared to encourage a larger community to collaborate on developing algorithms that answer visual questions [6, 7, 16, 17, 19, 20, 23, 28, 32, 39, 40, 43, 45].”
Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering (2016, methodology): “uilding trust for machines among their users. 2. Related Work Visual Question Answering. A number of recent works have proposed visual question answering datasets [3,21,25, 30,10,45,37,35] and models [9,24,2,42,23,26,46, 44,43,40,34,19,28,14,41,32,16]. Our work builds on top of the VQA dataset from ”
Analyzing the Behavior of Visual Question Answering Models (2016, background): “…have been proposed for VQA (Antol et al., 2015; Chen et al., 2015; Yang et al., 2015; Xu and Saenko, 2015; Jiang et al., 2015; Andreas et al., 2015; Wang et al., 2015; Kaﬂe and Kanan, 2016; Lu et al., 2016; Andreas et al., 2016; Shih et al., 2015; Kim et al., 2016; Fukui et al., 2016; Noh and…”
Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding (2018, neutral): “There have been numerous papers on VQA, among which some explicitly used structural knowledge to help reasoning [Wang et al., 2017].”
Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering (2017, background): “has received a lot of attention – a number of VQA datasets have been curated [5, 21, 41, 12, 25, 11, 29, 13, 39] and a variety of deep-learning models have been developed [5, 9, 38, 37, 16, 3, 34, 19, 24, 4, 31, 20, 10, 26, 14, 35, 36, 40, 30].”
Beyond Bilinear: Generalized Multimodal Factorized High-Order Pooling for Visual Question Answering (2017, background): “However, existing approaches have either only been applied to specific data sets [18] or have been ineffective on benchmark data sets [19].”
FVQA: Fact-Based Visual Question Answering (2016, methodology): “[51] as they described a method for visual question answering which is capable of reasoning about an image base on the information extracted from a knowledge base.”
Multi-modal Factorized Bilinear Pooling with Co-attention Learning for Visual Question Answering (2017, background): “However, existing approaches have either only been applied to speciﬁc datasets [29, 30], or have been ineffective on benchmark datasets [31].”
See, Think, Confirm: Interactive Prompting Between Vision and Language Models for Knowledge-based Visual Reasoning (2023, unknown): “Most early methods [20,23,29,40,58,71,76] use deep neural networks to understand images and retrieve relevant knowledge from explicit knowledge bases.”
Explicit Reasoning over End-to-End Neural Architectures for Visual Question Answering (2018, background): “Lately, there have been attempts of creating QA datasets that solely comprises of questions that require additional background knowledge along with information from images (Wang et al. 2015).”
Multimodal Intelligence: Representation Learning, Information Fusion, and Applications (2019, background): “longer trainable in an end-to-end manner [243], [244].”
Weakly-Supervised Visual-Retriever-Reader for Knowledge-based Question Answering (2021, methodology): “KB-VQA (Wang et al., 2017b) dataset consists of three types of questions: “Visual" question an-swerable using the visual concept in an image, “Common-sense" questions answerable by adults without looking for an external source, and “KB-knowledge" questions requiring higher-level knowledge, explicit ”
Visual question answering: A survey of methods and datasets (2016, methodology): “no need for being available at training time [87, 78].”
REVIVE: Regional Visual Representation Matters in Knowledge-Based Visual Question Answering (2022, background): “Recent studies [29, 28, 20, 19, 35, 31, 17, 32, 5] integrate different knowledge from various external knowledge resources, e.”
KnowIT VQA: Answering Knowledge-Based Questions about Videos (2019, methodology): “Nevertheless, KnowIT VQA with 24k questions is the largest KBVQA human-generated dataset, far from the 2.4k questions in KB-VQA, 5.8k in FVQA, and 14k in OK-VQA.”