The VQA-Machine: Learning How to Use Existing Vision Algorithms to Answer New Questions

By Peng Wang, Qi Wu, Chunhua Shen, Anton van den Hengel. 2016

90 citations recorded by the source corpus.

One of the most intriguing features of the Visual Question Answering (VQA) challenge is the unpredictability of the questions. Extracting the information required to answer them demands a variety of image operations from detection and counting, to segmentation and reconstruction. To train a method to perform even one of these operations accurately from {image, question, answer} tuples would be challenging, but to aim to achieve them all with a limited set of such training data seems ambitious at best. Our method thus learns how to exploit a set of external off-the-shelf algorithms to achieve its goal, an approach that has something in common with the Neural Turing Machine [10]. The core of our proposed method is a new co-attention model. In addition, the proposed approach generates human-readable reasons for its decision, and can still be trained end-to-end without ground truth reasons being given. We demonstrate the effectiveness on two publicly available datasets, Visual Genome and VQA, and show that it produces the state-of-the-art results in both cases.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Explicit Knowledge-based Reasoning for Visual Question Answering (2015, methodology): “[Wang et al., 2017] proposed a VQA method that performs reasoning on structured representation of images, which, however, does not utilize external structured KBs.”
Learning by Abstraction: The Neural State Machine (2019, methodology): “he need in this case for costly state updates, as in [46, 73]. Finally, we explore our model in the context of VQA, a challenging multimodal task that has gained substantial attention in recent years [25, 76, 38]. Prior work commonly relied on dense visual features produced by either CNNs [80, 83] o”
Beyond Short-Term Snippet: Video Relation Detection With Spatio-Temporal Global Context (2020, background): “Effectively capturing spatial and action relations between objects in a video is especially useful, improving the results in captioning [43, 45], video retrieval [2, 13], visual question answer [1, 39], and many other visual-language tasks.”
Recent Advances and Trends in Multimodal Deep Learning: A Review (2021, methodology): “[153] 2017 CNN/VGG19-ResNet100, RNN/LSTM Image, Text Visual Genome QA, VQA-real WUPS, Accuracy Z.”
Co-attending Free-form Regions and Detections with Multi-modal Multiplicative Feature Embedding for Visual Question Answering (2017, unknown): “9.11 36.41 46.42 - MRN (Kim et al. 2016) 61.68 82.28 38.82 49.25 66.15 61.84 82.39 38.23 49.41 66.33 HieCoAtt (Lu et al. 2016) 61.80 79.70 38.70 51.70 65.80 62.06 79.95 38.22 51.95 66.07 VQA-Machine (Wang et al. 2017) 63.10 81.50 38.40 53.00 67.70 63.30 81.40 38.20 53.20 67.80 MCB (Fukui et al. 2016”
Multi-Modal Sarcasm Detection in Twitter with Hierarchical Fusion Model (2019, methodology): “Following (Wang et al., 2017), a input image I is re-sized to 448 × 448 and divided into 14 × 14 regions.”
Relation-Aware Graph Attention Network for Visual Question Answering (2019, methodology): “To make the model more interpretable, some literature [30, 57, 29, 52, 53, 51] also exploited high-level semantic information in the image, such as attributes, captions and visual relation facts.”
Visual Grounding Via Accumulated Attention (2020, methodology): “Recently, the attention mechanism has been widely used in many applications such as Natural Language Processing [2, 23], Image Captioning [29, 30] and VQA [4, 14, 16, 24] due to its effectiveness in dealing with information redundancies.”
Visual Question Answering as a Meta Learning Task (2017, background): “from advanced attention mechanisms or other computer vision techniques [3, 37, 39].”
Can you fool AI with adversarial examples on a visual Turing test? (2017, background): “Some works argue that to successfully address VQA tasks one requires common sense knowledge which can be obtained from external knowledge bases [52], [51].”