FVQA: Fact-Based Visual Question Answering

By Peng Wang, Qi Wu, Chunhua Shen, Anthony R. Dick, Anton van den Hengel. 2016

550 citations recorded by the source corpus.

Visual Question Answering (VQA) has attracted much attention in both computer vision and natural language processing communities, not least because it offers insight into the relationships between two important sources of information. Current datasets, and the models built upon them, have focused on questions which are answerable by direct analysis of the question and image alone. The set of such questions that require no external information to answer is interesting, but very limited. It excludes questions which require common sense, or basic factual knowledge to answer, for example. Here we introduce FVQA (Fact-based VQA), a VQA dataset which requires, and supports, much deeper reasoning. FVQA primarily contains questions that require external information to answer. We thus extend a conventional visual question answering dataset, which contains image-question-answer triplets, through additional image-question-answer-supporting fact tuples. Each supporting-fact is represented as a structural triplet, such as <Cat,CapableOf,ClimbingTrees>. We evaluate several baseline models on the FVQA dataset, and describe a novel model which is capable of reasoning about an image on the basis of supporting-facts.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Towards VQA Models That Can Read (2019, methodology): “[45] introduced a Fact-Based VQA dataset which explicitly requires external knowledge to answer a question.”
KAT: A Knowledge Augmented Transformer for Vision-and-Language (2021, methodology): “Several methods explore aggregating the external knowledge either in the form of structured knowledge graphs (Garderes et al., 2020; Narasimhan et al., 2018; Li et al., 2020b; Wang et al., 2017a,b) or unstructured knowledge bases (Marino et al., 2021; Wu et al., 2021; Luo et al., 2021).”
An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA (2021, background): “Previous studies (Wang et al. 2015; Narasimhan and Schwing 2018; Wang et al. 2017; Narasimhan, Lazebnik, and Schwing 2018; Zhu et al. 2020; Li, Wang, and Zhu 2020; Marino et al. 2021; Wu et al. 2021) proposed various ways of retrieving and using the knowledge, and considered it necessary to use mult”
OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge (2019, methodology): “We also stem the answers using Porter stemming to consolidate answers that are identical except for pluralization and conjugation as in [37].”
VizWiz Grand Challenge: Answering Visual Questions from Blind People (2018, methodology): “0: Real [17] MSCOCO [25] Crowd workers (AMT) Typed Visual7W [45] MSCOCO [25] Crowd workers (AMT) Typed CLEVR [19] Synthetic Shapes Automatically generated (templates) ——– SHAPES [6] Synthetic Shapes Automatically generated (templates) ——– Visual Genome [23] MSCOCO [25] & YFCC100M [35] Crowd workers ”
From Recognition to Cognition: Visual Commonsense Reasoning (2018, background): “Recent datasets also focus on knowledge-base style content [80, 83].”
IQA: Visual Question Answering in Interactive Environments (2017, background): “These include datasets of natural images [2, 36, 41, 68], synthetic images [2, 4, 24, 27, 28], natural videos [23, 65], synthetic videos [32, 49] and multimodal contexts [29].”
Multi-modal Factorized Bilinear Pooling with Co-attention Learning for Visual Question Answering (2017, background): “However, existing approaches have either only been applied to speciﬁc datasets [29, 30], or have been ineffective on benchmark datasets [31].”
Graph R-CNN for Scene Graph Generation (2018, background): “These scene graphs form an interpretable structured representation of the image that can support higher-level visual intelligence tasks such as captioning [24,39], visual question answering [1,11,35,37–39], and image-grounded dialog [3].”
An Empirical Study on Leveraging Scene Graphs for Visual Question Answering (2019, unknown): “l be abstractly represented by a set of object entities and their relationships. Some Visual QA works indeed extract scene graphs followed by applying algorithms designed for reasoning on curated KBs [40,64]. In our work we also exploit scene graphs, but apply the recently published graph networks (”
Recent Advances and Trends in Multimodal Deep Learning: A Review (2021, unknown): “0 [160], VQA-CP [6], VQA-X [64], Ok-VQA [95], and FVQA [151].”
MemexQA: Visual Memex Question Answering (2017, background): “in [2] and has received a large amount of attention in recent years [31, 28, 14, 15, 8, 30, 27, 23].”
The Shallow End: Empowering Shallower Deep-Convolutional Networks through Auxiliary Outputs (2016, background): “Moreover, deep CNNs have also become the workhorse of many other tasks and real-world applications beyond computer vision, such as natural language understanding [5], [50] and speech recognition [24].”
Explicit Reasoning over End-to-End Neural Architectures for Visual Question Answering (2018, methodology): “has the form 〈subject, predicate, object〉 (Wang et al. 2016).”
Multi-Modal Knowledge Graph Construction and Application: A Survey (2022, background): “VQA aims at generating a textual answer for a textual question with the help of a relevant image [32], [33].”
REVIVE: Regional Visual Representation Matters in Knowledge-Based Visual Question Answering (2022, background): “OK-VQA dataset [18] is the first largescale dataset with questions that need be answered using external knowledge instead of a provided fixed knowledge base [28].”
From Two Graphs to N Questions: A VQA Dataset for Compositional Reasoning on Vision and Commonsense (2019, methodology): “FVQA [39] proposes to provide one type of additional annotations, that is, the knowledge triplet used to answer a question.”
Zero-Shot Recognition via Semantic Embeddings and Knowledge Graphs (2018, background): “Researchers have proposed several approaches on how to use knowledge graphs for object recognition [12, 43, 30, 35, 39, 9, 8, 29, 49, 47, 26].”