CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning

By Justin Johnson, Bharath Hariharan, L. V. D. van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, Ross B. Girshick. 2016

2,931 citations recorded by the source corpus.

When building artificial intelligence systems that can reason and answer questions about visual data, we need diagnostic tests to analyze our progress and discover short-comings. Existing benchmarks for visual question answering can help, but have strong biases that models can exploit to correctly answer questions without reasoning. They also conflate multiple sources of error, making it hard to pinpoint model weaknesses. We present a diagnostic dataset that tests a range of visual reasoning abilities. It contains minimal biases and has detailed annotations describing the kind of reasoning each question requires. We use this dataset to analyze a variety of modern visual reasoning systems, providing novel insights into their abilities and limitations.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

Learning Transferable Visual Models From Natural Language Supervision (2021, methodology): “…PatchCamelyon (Veeling et al., 2018), the UCF101 action recognition dataset (Soomro et al., 2012), Kinetics 700 (Carreira et al., 2019), 2,500 random samples of the CLEVR dataset (Johnson et al., 2017), the Hateful Memes dataset (Kiela et al., 2020), and the ImageNet-1k dataset (Deng et al., 2012).”
Towards a Better Metric for Evaluating Question Generation Systems (2018, background): “…Answering (QA) (Rajpurkar et al., 2016; Nguyen et al., 2016; Joshi et al., 2017; Saha et al., 2018a) knowledge base driven QA (Bordes et al., 2015; Saha et al., 2018b) and Visual QA (Antol et al., 2015; Johnson et al., 2017) has enabled the development of end-to-end supervised models for QA.”
Break It Down: A Question Understanding Benchmark (2020, methodology): “Kwiatkowski et al. (2013) and Choi et al. (2015) used underspecified logical forms as an intermediate representation.”
On Robustness and Transferability of Convolutional Neural Networks (2020, background): “The structured group contains the following tasks: counting and distance prediction on CLEVR [33], pixel-location and orientation prediction on dSprites [45], camera elevation and object orientation on SmallNORB [41], object distance on DMLab [4] and vehicle distance on KITTI [19].”
Object-Centric Learning with Slot Attention (2020, methodology): “[75] Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick.”
A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark (2019, background): “Patch Camelyon (Veeling et al., 2018) Clevr/count (Johnson et al., 2017) CLEVR is a visual question and answer dataset designed to evaluate algorithmic visual reasoning.”
The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes (2020, background): “[56] for a survey), with great progress on important problems such as visual question answering [3, 26] and image caption generation and retrieval [8, 88, 43, 68, 28], with offshoot tasks focusing specifically on visual reasoning [36], referring expressions [39], visual storytelling [57, 32], visual”
Towards VQA Models That Can Read (2019, methodology): “To study visual reasoning independent of language, non-photo-realistic VQA datasets have been introduced such as CLEVR [18], NLVR [42] and FigureQA [21].”
Learning with Latent Language (2017, neutral): “These kinds of reasoning problems have been well-studied in visual question answering settings (Johnson et al., 2017; Suhr et al., 2017).”
ShapeWorld - A new test methodology for multimodal language understanding (2017, methodology): “Most similar to the S HAPE W ORLD framework is the CLEVR dataset (Johnson et al., 2017).”
A simple neural network module for relational reasoning (2017, methodology): “The ability to reason about the relations between entities and their properties is central to generally intelligent behavior (Figure 1) [10, 7].”
Grammar-Based Grounded Lexicon Learning (2022, methodology): “We first evaluate G2L2 on the visual reasoning tasks in the CLEVR domain [23], where the task is to reason and answer questions about images.”
ALFWorld: Aligning Text and Embodied Environments for Interactive Learning (2020, background): “Vision and language : While substantial work exists on vision-language representation learning e.g., MAttNet (Yu et al., 2018b), CMN (Hu et al., 2017), VQA (Antol et al., 2015), CLEVR (Johnson et al., 2017), ViLBERT (Lu et al., 2019), they lack embodied or sequential decision making.”
Discovering Language Model Behaviors with Model-Written Evaluations (2022, methodology): “Other work automatically constructs evaluations not with LMs but using example templates or programmatic generation (Weston et al., 2016; Johnson et al., 2017; McCoy et al., 2019).”
Multiset-Equivariant Set Prediction with Approximate Implicit Differentiation (2021, methodology): “Table 9 and Table 10 show predictions on CLEVR for different number of evaluation iterations (Section 4.3).”