Visual Question Answering as a Meta Learning Task

By Damien Teney, Anton van den Hengel. 2017

45 citations recorded by the source corpus.

The predominant approach to Visual Question Answering (VQA) demands that the model represents within its weights all of the information required to answer any question about any image. Learning this information from any real training set seems unlikely, and representing it in a reasonable number of weights doubly so. We propose instead to approach VQA as a meta learning task, thus separating the question answering method from the information required. At test time, the method is provided with a support set of example questions/answers, over which it reasons to resolve the given question. The support set is not fixed and can be extended without retraining, thereby expanding the capabilities of the model. To exploit this dynamically provided information, we adapt a state-of-the-art VQA model with two techniques from the recent meta learning literature, namely prototypical networks and meta networks. Experiments demonstrate the capability of the system to learn to produce completely novel answers (i.e. never seen during training) from examples provided at test time. In comparison to the existing state of the art, the proposed method produces qualitatively distinct results with higher recall of rare answers, and a better sample efficiency that allows training with little initial data. More importantly, it represents an important step towards vision-and-language methods that can learn and reason on-the-fly.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

REMIND Your Neural Network to Prevent Catastrophic Forgetting (2019, background): “REMIND could also be used to improve VQA models that exploit additional data during test time to answer new questions [63].”
Hierarchical Graph Attention Network for Few-shot Visual-Semantic Learning (2021, methodology): “[46] obtained their model input through a simple element-wise production between visual and semantic representations.”
MetaReg: Towards Domain Generalization using Meta-Regularization (2018, background): “Other recent applications that use meta learning include imitation learning [8], visual question answering [29], etc.”
Answer Them All! Toward Universal Visual Question Answering Models (2019, methodology): “Many algorithms for natural image VQA have been proposed, including Bayesian approaches [23, 36], methods using spatial attention [52, 33, 40, 6], compositional approaches [7, 8, 18], bilinear pooling schemes [29, 14], and others [50, 41, 26].”
Compact Trilinear Interaction for Visual Question Answering (2019, methodology): “There are different approaches have been proposed for VQA [18, 5, 8, 45, 20, 42, 24, 2, 28, 23, 38, 46, 29, 40].”