GQA: a new dataset for compositional question answering over real-world images

By Drew A. Hudson, Christopher D. Manning. 2019

155 citations recorded by the source corpus.

We introduce GQA, a new dataset for real-world visual reasoning and compositional question answering, seeking to address key shortcomings of previous VQA datasets. We have developed a strong and robust question engine that leverages scene graph structures to create 22M diverse reasoning questions, all come with functional programs that represent their semantics. We use the programs to gain tight control over the answer distribution and present a new tunable smoothing technique to mitigate language biases. Accompanying the dataset is a suite of new metrics that evaluate essential qualities such as consistency, grounding and plausibility. An extensive analysis is performed for baselines as well as state-of-the-art models, providing fine-grained results for different question types and topologies. Whereas a blind LSTM obtains mere 42.1%, and strong VQA models achieve 54.1%, human performance tops at 89.3\%, offering ample opportunity for new research to explore. We strongly hope GQA will provide an enabling resource for the next generation of models with enhanced robustness, improved consistency, and deeper semantic understanding for images and language.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Question Answering & Reading Comprehension

Citations in the corpus

LXMERT: Learning Cross-Modality Encoder Representations from Transformers (2019, methodology): “VQA The SotA result is BAN+Counter in Kim et al. (2018), which achieves the best accuracy among other recent works: MFH (Yu et al., 2018), Pythia (Jiang et al., 2018), DFAF (Gao et al., 2019a), and Cycle-Consistency (Shah et al., 2019).5 LXMERT improves the SotA overall accuracy (‘Accu’ in Table 2) ”
MAGMA - Multimodal Augmentation of Generative Models through Adapter-based Finetuning (2021, methodology): “…of structured, task-based datasets [42,56], we also opt to include the training sets of various downstream tasks – adding the train sets of VQA [2], GQA [23], OKVQA [31], VizWiz [17], Hateful Memes [25], and CoCo Captions [9], resulting in a ﬁnal dataset of around 25 million image-text pairs.”
Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts (2021, background): “Some work [72, 21, 47] use V+L resources collected for dense captioning or visual question answering (VQA), such as VG [41], VQA2 [26], and GQA [33].”
A Corpus for Reasoning about Natural Language Grounded in Photographs (2018, background): “, 2017), which uses human-written captions that compare and contrast sets of synthetic images; and GQA (Hudson and Manning, 2019), which uses synthetic language grounded in real-world photographs.”
Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Models (2020, background): “stream tasks, these pre-trained models have achieved new state of the art on image-text retrieval [18], visual question answering [4,11], referring expression comprehension [37], and visual reasoning [13,28,38]. This suggests that substantial amount of visual and linguistic knowledge has been encode”
Are Red Roses Red? Evaluating Consistency of Question-Answering Models (2019, methodology): “Since questions are still synthetic, and “not as natural as other VQA datasets” (Hudson and Manning, 2019), it remains to be seen whether models will overfit to the generation procedure or to the implications encoded (e.”
An Empirical Study on Leveraging Scene Graphs for Visual Question Answering (2019, methodology): “Nevertheless, our model can be easily extended into the open-ended setting with the answer embedding strategy [28].”
Shifting the Baseline: Single Modality Performance on Visual Navigation & QA (2018, background): “Questions in VQA are conceptually simple, but in theory require understanding the relationships between different objects in the scene to answer correctly.”
Iterative Answer Prediction With Pointer-Augmented Multimodal Transformers for TextVQA (2019, unknown): “[4, 17, 22, 21, 20]) and methods ( e.g .”
Language-Conditioned Graph Networks for Relational Reasoning (2019, methodology): “We first experiment with using the released object detection features in the GQA dataset as our local features x, which is shown in [19] to perform better than the convolutional grid features, and compare with previous works.”
Pre-training image-language transformers for open-vocabulary tasks (2022, background): “0 [1], GQA [4], visual entailment (SNLI-VE [18]) and captioning (MSCOCO [2]).”
Multi-Modal Answer Validation for Knowledge-Based VQA (2021, background): “Over the past few years, the domain of Visual Question Answering (VQA) has witnessed signiﬁcant progress (Antol et al. 2015; Zhu et al. 2016; Hudson and Manning 2019).”
InfographicVQA (2021, background): “VQA received signiﬁcant attention over the past few years [17, 5, 18, 22, 27, 3].”
Video2Commonsense: Generating Commonsense Descriptions to Enrich Video Captioning (2020, background): “Spatial and compositional reasoning is required to answer questions about synthetic images in CLEVR[15] and about natural images in GQA[14].”
Decoupled Box Proposal and Featurization with Ultrafine-Grained Semantic Labels Improve Image Captioning and Visual Question Answering (2019, background): “1 (Jiang et al., 2018) to preprocess VizWiz dataset and retain 3135 top answers.”
From Two Graphs to N Questions: A VQA Dataset for Compositional Reasoning on Vision and Commonsense (2019, methodology): “Our dataset proposes a more general form of the questions to unify the visual questions [29, 33, 15, 19] and the commonsense questions [39] in some datasets, and to provide more challenging compositional commonsense questions which are unique for existing datasets.”
Violin: A Large-Scale Dataset for Video-and-Language Inference (2020, unknown): “5K VCR [72] image movie 7 3 QA 110K 290K GQA [25] image indoor 7 3 QA 113K 22M SNLI-VE [61] image natural 7 3 Entailment 31.”