Yin and Yang: Balancing and Answering Binary Visual Questions

By Peng Zhang, Yash Goyal, Douglas Summers-Stay, Dhruv Batra, Devi Parikh. 2015

373 citations recorded by the source corpus.

The complex compositional structure of language makes problems at the intersection of vision and language challenging. But language also provides a strong prior that can result in good superficial performance, without the underlying models truly understanding the visual content. This can hinder progress in pushing state of art in the computer vision aspects of multi-modal AI. In this paper, we address binary Visual Question Answering (VQA) on abstract scenes. We formulate this problem as visual verification of concepts inquired in the questions. Specifically, we convert the question to a tuple that concisely summarizes the visual concept to be detected in the image. If the concept can be found in the image, the answer to the question is "yes", and otherwise "no". Abstract scenes play two roles (1) They allow us to focus on the highlevel semantics of the VQA task as opposed to the low-level recognition problems, and perhaps more importantly, (2) They provide us the modality to balance the dataset such that language priors are controlled, and the role of vision is essential. In particular, we collect fine-grained pairs of scenes for every question, such that the answer to the question is "yes" for one scene, and "no" for the other for the exact same question. Indeed, language priors alone do not perform better than chance on our balanced dataset. Moreover, our proposed approach matches the performance of a state-of-the-art VQA approach on the unbalanced dataset, and outperforms it on the balanced dataset.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

ShapeWorld - A new test methodology for multimodal language understanding (2017, background): “Abstract Clipart scenes have been used for image captioning (Zitnick et al., 2016; Zitnick and Parikh, 2013) and to balance existing VQA datasets (Zhang et al., 2016).”
Unified Vision-Language Pre-Training for Image Captioning and VQA (2019, background): “…model architectures to fuse question and image representations (Kim, Jun, and Zhang 2018; Anderson et al. 2018; Gao et al. 2019), new datasets or models to reduce the dataset bias (Zhang et al. 2016; Goyal et al. 2017; Agrawal et al. 2017) and ground the answer in the question (Lewis and Fan 2019).”
Visual Dialog (2016, methodology): “Binary answers in VQA are biased towards ‘yes’ [4,61] – 61.”
Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering (2022, neutral): “Since the task of visual question answering (VQA) was ﬁrst proposed in [2], there have been plenty of VQA datasets [55, 57, 22, 10, 14, 11] conducted to facilitate the research work.”
GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering (2019, background): “Multiple attempts have been made to mitigate the systematic biases of VQA datasets as discussed in section 1 [9, 34, 2, 13], but they fall short in providing an adequate solution: Some approaches operate over constrained and synthetic images [34, 13], neglecting the realism and diversity natural pho”
A Corpus for Reasoning about Natural Language Grounded in Photographs (2018, methodology): “This is re-lated to recent visual question answering datasets that aim to require models to consider both image and question to perform well (Zhang et al., 2016; Goyal et al., 2017; Li et al., 2017; Agrawal et al., 2017, 2018).”
Fusion of Detected Objects in Text for Visual Question Answering (2019, background): “VQA (Antol et al., 2015; Zhang et al., 2016; Goyal et al., 2017) contains photos and abstract scenes with questions and several ground-truth answers for each, but the questions are less complex than VCR’s.”
Hierarchical Question-Image Co-Attention for Visual Question Answering (2016, background): “Many recent works [2, 6, 11, 14, 15, 25] have proposed models for VQA.”
Inferring and Executing Programs for Visual Reasoning (2017, background): “Recent work has questioned whether such systems are capable of developing visual reasoning capabilities: (1) very simple baseline models were found to perform competitively on VQA benchmarks by exploiting biases in the data [18, 49, 11] and (2) experiments on CLEVR, which was designed to control suc”
Analyzing the Behavior of Visual Question Answering Models (2016, background): “Label biases (in particular, for “yes/no” questions) have also been reported in (Zhang et al., 2016).”
Embodied Question Answering (2017, positive): “This enables us to control the distribution of question-types and answers, deter algorithms from exploiting dataset bias [4, 9], and provide ﬁne-grained breakdown of performance by skill.”
Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering (2017, background): “However, a number of studies have found that despite recent progress, today’s VQA models are heavily driven by superficial correlations in the training data and lack sufficient visual grounding [8,9,28,29].”
Visual question answering: Datasets, algorithms, and future challenges (2016, methodology): “In [42], bias in VQA was studied using synthetic cartoon images.”
An Analysis of Visual Question Answering Algorithms (2017, methodology): “Six datasets for VQA with natural images have been released between 2014–2016: DAQUAR [21], COCOQA [25], FM-IQA [6], The VQA Dataset [3], Visual7W [35], and Visual Genome [18].”
FVQA: Fact-Based Visual Question Answering (2016, neutral): “1 VQA-balanced [67] 15,623 33,379 1 6.”