Ask Me Anything: Free-Form Visual Question Answering Based on Knowledge from External Sources
By Qi Wu, Peng Wang, Chunhua Shen, Anthony R. Dick, Anton van den Hengel. 2015
392 citations recorded by the source corpus.
We propose a method for visual question answering which combines an internal representation of the content of an image with information extracted from a general knowledge base to answer a broad range of image-based questions. This allows more complex questions to be answered using the predominant neural network-based approach than has previously been possible. It particularly allows questions to be asked about the contents of an image, even when the image itself does not contain the whole answer. The method constructs a textual representation of the semantic content of an image, and merges it with textual information sourced from a knowledge base, to develop a deeper understanding of the scene viewed. Priming a recurrent neural network with this combined information, and the submitted question, leads to a very flexible visual question answering approach. We are specifically able to answer questions posed in natural language, that refer to information not contained in the image. We demonstrate the effectiveness of our model on two publicly available datasets, Toronto COCO-QA [23] and VQA [1] and show that it produces the best reported results in both cases.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Retrieval-Augmented Generation, Multimodal Learning, Question Answering & Reading Comprehension
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Dacheng Tao
- Trevor Darrell
- Yoshua Bengio
- Devi Parikh
- Yejin Choi
- Percy Liang
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Yi Yang
- Ming-Hsuan Yang
- Jitendra Malik
- Cordelia Schmid
In Retrieval-Augmented Generation
- Anton van den Hengel
- Christopher Kanan
- Kushal Kafle
- Damien Teney
- Chunhua Shen
- Qi Wu
- Peng Wang
- Anthony R. Dick
In Multimodal Learning
- Devi Parikh
- Anton van den Hengel
- Christopher Kanan
- Qi Wu
- Kushal Kafle
- Marcus Rohrbach
- Dhruv Batra
- Mohit Bansal
In Question Answering & Reading Comprehension
- Christopher Kanan
- Anton van den Hengel
- Kushal Kafle
- Damien Teney
- Devi Parikh
- Qi Wu
- Yueting Zhuang
- Hanwang Zhang
Citations in the corpus
- OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge (2019, background): “The problem of using external knowledge for answering questions has been tackled by [45, 43, 44, 29, 36, 35].”
- Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering (2016, methodology): “uilding trust for machines among their users. 2. Related Work Visual Question Answering. A number of recent works have proposed visual question answering datasets [3,21,25, 30,10,45,37,35] and models [9,24,2,42,23,26,46, 44,43,40,34,19,28,14,41,32,16]. Our work builds on top of the VQA dataset from ”
- From Recognition to Cognition: Visual Commonsense Reasoning (2018, background): “Recent datasets also focus on knowledge-base style content [80, 83].”
- Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering (2015, methodology): “In this paper, we explore a complementary approach of spatial attention to both improve performance and visualize the network’s inference process, and obtain improved results without using external data compared to the iBOWIMG model [3] as well as the ACK model [26] and the DPPnet model [27] which u”
- Analyzing the Behavior of Visual Question Answering Models (2016, background): “A flurry of recent deep-learning based models have been proposed for VQA (Antol et al., 2015; Chen et al., 2015; Yang et al., 2016; Xu and Saenko, 2016; Jiang et al., 2015; Andreas et al., 2016a; Wang et al., 2015; Kafle and Kanan, 2016; Lu et al., 2016; Andreas et al., 2016b; Shih et al., 2016; Kim”
- MUTAN: Multimodal Tucker Fusion for Visual Question Answering (2017, neutral): “ teststd Y/N No. Other All All SMem 2-hop [27] 80.87 37.32 43.12 57.99 58.24 Ask Your Neur. [20] 78.39 36.45 46.28 58.39 58.43 SAN [30] 79.3 36.6 46.1 58.7 58.9 D-NMN [1] 81.1 38.6 45.5 59.4 59.4 ACK [26] 81.01 38.42 45.23 59.17 59.44 MRN [7] 82.28 38.82 49.25 61.68 61.84 HieCoAtt [17] 79.7 38.7 51.”
- Simple Baseline for Visual Question Answering (2015, methodology): “Open-Ended Overall yes/no number others LSTMIMG [2] 54.06 - - - NMN+LSTM [1] 55.10 - - - ACK [16] 55”
- Learning Language-Visual Embedding for Movie Understanding with Natural-Language (2016, background): “There are also abundant evidence [19,27] that jointly learning from language and visual modalities can be mutually beneficial.”
- Dual Attention Networks for Multimodal Reasoning and Matching (2016, neutral): “7 ACK [31] 81.”
- Explicit Knowledge-based Reasoning for Visual Question Answering (2015, methodology): “[Wu et al., 2016] encode the mined knowledge text from the DBpedia to a vector and combined with visual features together to generate answers using an LSTM model.”
- Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering (2017, background): “has received a lot of attention – a number of VQA datasets have been curated [5, 21, 41, 12, 25, 11, 29, 13, 39] and a variety of deep-learning models have been developed [5, 9, 38, 37, 16, 3, 34, 19, 24, 4, 31, 20, 10, 26, 14, 35, 36, 40, 30].”
- Beyond Bilinear: Generalized Multimodal Factorized High-Order Pooling for Visual Question Answering (2017, background): “However, existing approaches have either only been applied to specific data sets [18] or have been ineffective on benchmark data sets [19].”
- Visual question answering: Datasets, algorithms, and future challenges (2016, methodology): “Consistent with other papers [36, 37, 38], the rest of The VQA Dataset will be referred as COCO-VQA, since it contains images from the COCO image dataset.”
- An Analysis of Visual Question Answering Algorithms (2017, methodology): “We only use Visual Genome to make color and positional reasoning questions.”
- FVQA: Fact-Based Visual Question Answering (2016, methodology): “Neither [59] nor [60] are capable of explicit reasoning, in contrast to the method we propose here.”