Zero-Shot Visual Question Answering

By Damien Teney, Anton van den Hengel. 2016

75 citations recorded by the source corpus.

Part of the appeal of Visual Question Answering (VQA) is its promise to answer new questions about previously unseen images. Most current methods demand training questions that illustrate every possible concept, and will therefore never achieve this capability, since the volume of required training data would be prohibitive. Answering general questions about images requires methods capable of Zero-Shot VQA, that is, methods able to answer questions beyond the scope of the training questions. We propose a new evaluation protocol for VQA methods which measures their ability to perform Zero-Shot VQA, and in doing so highlights significant practical deficiencies of current approaches, some of which are masked by the biases in current datasets. We propose and evaluate several strategies for achieving Zero-Shot VQA, including methods based on pretrained word embeddings, object classifiers with semantic embeddings, and test-time retrieval of example images. Our extensive experiments are intended to serve as baselines for Zero-Shot VQA, and they also achieve state-of-the-art performance in the standard VQA evaluation setting.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Question Answering & Reading Comprehension

Citations in the corpus

Just Ask: Learning to Answer Questions from Millions of Narrated Videos (2020, neutral): “To evaluate the generalization of VQA models, Teney and Hengel [75] define zero-shot VQA by answering previously unseen questions, which is a related but less challenging task compared to the zero-shot VQA task we propose in Section 6.”
Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering (2017, methodology): “This approach has been the basis of numerous previous models [16, 20, 38].”
Tips and Tricks for Visual Question Answering: Learnings from the 2017 Challenge (2017, methodology): “This is consistent with ﬁndings previously reported in [31].”
Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering (2017, background): “Zero-shot VQA has also been explored in [33].”
Learning What Makes a Difference from Counterfactual Examples and Gradient Supervision (2020, background): “Controlled evaluations for VQA now include out-of-distribution test sets [2,65].”
Separating Skills and Concepts for Novel Visual Question Answering (2021, unknown): “[56] creates testing splits such that at least one word of a question is un-Where”
Visual Question Answering as a Meta Learning Task (2017, methodology): “An increased interest has appeared in the handling of rare words and answers [29, 35].”
All You May Need for VQA are Image Captions (2022, background): “Note that the term zero-shot here is different from the one used in (Teney and Hengel, 2016), in which the model still has access to manually-created VQA triplets but is evaluated with unseen questions at test time.”
Bottom-Up and Top-Down Attention for Image Captioning and VQA (2017, methodology): “This approach has been the basis of numerous previous models [18, 23, 41].”
Compact Trilinear Interaction for Visual Question Answering (2019, methodology): “There are different approaches have been proposed for VQA [18, 5, 8, 45, 20, 42, 24, 2, 28, 23, 38, 46, 29, 40].”
A Zero-Shot Framework for Sketch-based Image Retrieval (2018, background): “Recently, zero shot learning has been gaining increasing attention for a number of other computer vision tasks such as image tagging [37,38], visual question answering [39,40] action recognition [41] etc.”