GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering

By Drew A. Hudson, Christopher D. Manning. 2019

3,277 citations recorded by the source corpus.

We introduce GQA, a new dataset for real-world visual reasoning and compositional question answering, seeking to address key shortcomings of previous VQA datasets. We have developed a strong and robust question engine that leverages Visual Genome scene graph structures to create 22M diverse reasoning questions, which all come with functional programs that represent their semantics. We use the programs to gain tight control over the answer distribution and present a new tunable smoothing technique to mitigate question biases. Accompanying the dataset is a suite of new metrics that evaluate essential qualities such as consistency, grounding and plausibility. A careful analysis is performed for baselines as well as state-of-the-art models, providing fine-grained results for different question types and topologies. Whereas a blind LSTM obtains a mere 42.1%, and strong VQA models achieve 54.1%, human performance tops at 89.3%, offering ample opportunity for new research to explore. We hope GQA will provide an enabling resource for the next generation of models with enhanced robustness, improved consistency, and deeper semantic understanding of vision and language.

Subfields: Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

How Can We Accelerate Progress Towards Human-like Linguistic Generalization? (2020, methodology): “…as a carte blanche to ignore sample efﬁciency, then, we should address it experimentally, by collecting multimodal data sets (Suhr et al., 2019; Hudson and Manning, 2019), developing models that learn from them efﬁciently, and using the Generalization Leaderboard to measure how effective this…”
UNITER: Learning UNiversal Image-TExt Representations (2019, methodology): “Note that LXMERT pre-trains with downstream VQA (+VG+GQA) data, which may help adapt the model to VQA task.”
A Multimodal Framework for the Detection of Hateful Memes (2020, background): “0 [10], GQA [12], and VG-QA [38].”
VirTex: Learning Visual Representations from Textual Annotations (2020, background): “Vision-and-Language Pretraining attempts to learn joint representations of image-text paired data that can be transferred to multimodal downstream tasks such as visual question answering [13, 14, 59, 60], visual reasoning [61, 62], referring expressions [63], and language-based image retrieval [35].”
The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes (2020, background): “In the past few years there has been a surge of interest in multimodal problems, from image captioning [8, 43, 88, 28, 68] to visual question answering (VQA) [3, 26, 27, 33, 71] and beyond.”
Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks (2020, methodology): “Method LXMERT MMN [5] 12-in-1 NSM [13] OscarB OscarB ⇤ Test-dev 60.”
Towards Language-Free Training for Text-to-Image Generation (2021, background): “These works aim at learning generic multi-modal representations for downstream tasks like visual question answering [2, 14], image captioning [1, 27], visual commonsense reasoning [55].”
Unsupervised Question Decomposition for Question Answering (2020, background): “Question answering (QA) systems have become remarkably good at answering simple, single-hop questions but still struggle with compositional, multi-hop questions (Yang et al., 2018; Hudson & Manning, 2019).”
STAR: A Benchmark for Situated Reasoning in Real-World Videos (2024, background): “It was first studied in CLEVR [17] and GQA [14] for reasoning in static images.”
VL-ADAPTER: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks (2021, methodology): “For image-text experiments, we evaluate our models on four V&L datasets: VQAv2 [10] and GQA [19]”
Achieving Human Parity on Visual Question Answering (2021, background): “0 [4], GQA balanced version [30], and VG-QA [99].”
Unifying Vision-and-Language Tasks via Text Generation (2021, methodology): “…approaches (Yu et al., 2018a; Anderson et al., 2018; Kim et al., 2018; Yu et al., 2018b) in a wide range of discriminative (Goyal et al., 2019; Hudson & Manning, 2019; Lei et al., 2018; Mao et al., 2016; Xu et al., 2016; Zhou et al., 2018) and generative tasks (Chen et al., 2015; Xu et al.,…”
VL-BERT: Pre-training of Generic Visual-Linguistic Representations (2019, background): “…noting that the input formats vary for different visual-linguistic tasks (e.g., < Caption, Image > for image captioning, and < Question, Answer, Image > for VQA (Antol et al., 2015; Johnson et al., 2017; Goyal et al., 2017; Hudson & Manning, 2019) and VCR (Zellers et al., 2019; Gao et al., 2019)).”
Improved Baselines with Visual Instruction Tuning (2023, methodology): “We first include We choose to conduct experiments on GQA [20], MME [16], and MM-Vet [54] to examine the representative capabilities of VQA with short answers, VQA with output formatting, and natural visual conversations, respectively.”
ViperGPT: Visual Inference via Python Execution for Reasoning (2023, methodology): “We use the GQA dataset [26], which was created to measure performance on complex compositional questions.”
BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models (2023, positive): “BLIP-2 achieves state-of-the-art result on the VQAv2 (Goyal et al., 2017) and GQA (Hudson & Manning, 2019) datasets.”
Visual Programming: Compositional visual reasoning without training (2022, methodology): “On NLVR, the performance of VISPROG saturates with fewer prompts than GQA.”