Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering
By Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, Devi Parikh. 2016
4,360 citations recorded by the source corpus.
The problem of visual question answering (VQA) is of significant importance both as a challenging research question and for the rich set of applications it enables. In this context, however, inherent structure in our world and bias in our language tend to be a simpler signal for learning than visual modalities, resulting in VQA models that ignore visual information, leading to an inflated sense of their capability. We propose to counter these language priors for the task of VQA and make vision (the V in VQA) matter! Specifically, we balance the popular VQA dataset (Antol et al., in: ICCV, 2015) by collecting complementary images such that every question in our balanced dataset is associated with not just a single image, but rather a pair of similar images that result in two different answers to the question. Our dataset is by construction more balanced than the original VQA dataset and has approximately twice the number of image-question pairs. Our complete balanced dataset is publicly available at http://visualqa.org/ as part of the 2nd iteration of the VQA Dataset and Challenge (VQA v2.0). We further benchmark a number of state-of-art VQA models on our balanced dataset. All models perform significantly worse on our balanced dataset, suggesting that these models have indeed learned to exploit language priors. This finding provides the first concrete empirical evidence for what seems to be a qualitative sense among practitioners. We also present interesting insights from analysis of the participant entries in VQA Challenge 2017, organized by us on the proposed VQA v2.0 dataset. The results of the challenge were announced in the 2nd VQA Challenge Workshop at the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2017. Finally, our data collection protocol for identifying complementary images enables us to develop a novel interpretable model, which in addition to providing an answer to the given (image, question) pair, also provides a counter-example based explanation. Specifically, it identifies an image that is similar to the original image, but it believes has a different answer to the same question. This can help in building trust for machines among their users.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Percy Liang
- Yoshua Bengio
- Luke S. Zettlemoyer
- Yejin Choi
- Quoc V. Le
- Caiming Xiong
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Ming-Hsuan Yang
- Jitendra Malik
- Cordelia Schmid
- Andrew Zisserman
In Multimodal Learning
Citations in the corpus
- SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference (2018, methodology): “Last, annotation artifacts are also a re-curring problem for vision datasets such as Visual Genome (Zellers et al., 2018) and Visual QA (Jabri et al., 2016); recent work was done to create a more challenging VQA dataset by annotating complementary image pairs (Goyal et al., 2016).”
- Adversarial NLI: A New Benchmark for Natural Language Understanding (2019, background): “2015), biases like “2” being the correct answer to 39% of the questions starting with “how many” allowed learning algorithms to perform well while ignoring the visual modality altogether (Jabri et al., 2016; Goyal et al., 2017).”
- Annotation Artifacts in Natural Language Inference Data (2018, background): “Agrawal et al. (2016); Jabri et al. (2016); Goyal et al. (2017) all showed that state-of-the-art visual question answering (Antol et al., 2015) systems leverage annotation biases in the dataset.”
- Shortcut learning in deep neural networks (2020, background): “f a sentence [14]. Natural Language Processing suffers from very similar problems as Computer Vision and other fields. Shortcut learning starts from various dataset biases such as annotation artefacts [55,56,57,58]. Feature combination crucially depends on shortcut features like word length [59,60,14”
- VisualBERT: A Simple and Performant Baseline for Vision and Language (2019, methodology): “We conduct comprehensive experiments on four vision-and-language tasks: (1) visual question answering (VQA 2.0, Goyal et al. (2017)), (2) visual commonsense reasoning (VCR, Zellers et Figure 1: Attention weights of some selected heads in VisualBERT.”
- A Multimodal Framework for the Detection of Hateful Memes (2020, neutral): “0 [10], GQA [12], and VG-QA [38].”
- Learning Visual Representations with Caption Annotations (2020, background): “Vision and language (VL) have been jointly leveraged to learn cross-modal representations for various VL tasks, such as crossmodal retrieval [21,61], visual question answering [25], captioning [56] or visual grounding [11,30].”
- VirTex: Learning Visual Representations from Textual Annotations (2020, background): “Vision-and-Language Pretraining attempts to learn joint representations of image-text paired data that can be transferred to multimodal downstream tasks such as visual question answering [13, 14, 59, 60], visual reasoning [61, 62], referring expressions [63], and language-based image retrieval [35].”
- Large-Scale Adversarial Training for Vision-and-Language Representation Learning (2020, background): “The training curves are provided in Figure 3b. Compared to LXMERT, V ILLA -fine achieves higher accuracy on validation set and lower accuracy on training set for both VQA and GQA, clearly demonstrating its regularization effect in preventing overfitting of large-scale models.”
- The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes (2020, background): “This strategy is compatible with recent work arguing that tasks should include counterfactual or contrastive examples [38, 22, 26].”
- LXMERT: Learning Cross-Modality Encoder Representations from Transformers (2019, background): “0 dataset (Goyal et al., 2017), GQA (Hudson and Manning, 2019), and NLVR 2 .”
- Towards VQA Models That Can Read (2019, methodology): “[10] by adding complementary triplets (Ic, q, ac) for each original triplet (Io, q, ao) where image Ic is similar to image Io but the answer for the given question q changes from ao to ac.”
- ShapeWorld - A new test methodology for multimodal language understanding (2017, methodology): “A S HAPE W ORLD dataset differs from standard VQA evaluation datasets in three main ways.”
- How Much Can CLIP Benefit Vision-and-Language Tasks? (2021, methodology): “For instance, if we cast VQA 2.0 (Goyal et al., 2017) into a zero-shot image-to-text retrieval task, we only observe chance performance (Section 5).”
- STAR: A Benchmark for Situated Reasoning in Real-World Videos (2024, background): “Various benchmarks [11, 44, 20] have been constructed by asking annotators to provide designed questions [1, 44] or synthesizing questions with pre-defined templates [15, 24, 14, 6].”
- BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation (2022, background): “0 dataset (Goyal et al., 2017), which contains 83k/41k/81k images for training/validation/test.”
- MERLOT RESERVE: Neural Script Knowledge through Vision and Language and Sound (2022, background): “A common pitfall when training a joint multimodal model is that complex inter-modal interactions can be ignored during learning, in favor of simpler intra-modal interactions [50, 23, 58].”