CLEVR-Dialog: A Diagnostic Dataset for Multi-Round Reasoning in Visual Dialog

By Satwik Kottur, José M. F. Moura, Devi Parikh, Dhruv Batra, Marcus Rohrbach. 2019

94 citations recorded by the source corpus.

Visual Dialog is a multimodal task of answering a sequence of questions grounded in an image (using the conversation history as context). It entails challenges in vision, language, reasoning, and grounding. However, studying these subtasks in isolation on large, real datasets is infeasible as it requires prohibitively-expensive complete annotation of the ‘state’ of all images and dialogs. We develop CLEVR-Dialog, a large diagnostic dataset for studying multi-round reasoning in visual dialog. Specifically, we construct a dialog grammar that is grounded in the scene graphs of the images from the CLEVR dataset. This combination results in a dataset where all aspects of the visual dialog are fully annotated. In total, CLEVR-Dialog contains 5 instances of 10-round dialogs for about 85k CLEVR images, totaling to 4.25M question-answer pairs. We use CLEVR-Dialog to benchmark performance of standard visual dialog models; in particular, on visual coreference resolution (as a function of the coreference distance). This is the first analysis of its kind for visual dialog models that was not possible without this dataset. We hope the findings from CLEVR-Dialog will help inform the development of future models for visual dialog. Our code and dataset are publicly available.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Question Answering & Reading Comprehension

Citations in the corpus

Vision-and-Dialog Navigation (2019, background): “Dataset —Language Context— —Visual Context— Human Amb UnderS Temporal Real-world Temporal Shared MARCO[8, 9], DRIF[10] 3 7 7 1I 7 Dynamic R2R[7], Touchdown[11] 3 7 7 1I 3 Dynamic EQA[12], IQA[13] 7 7 3 1Q 7 Dynamic CLEVR[14] 7 7 1Q 7 Static VQA[15, 16, 17] 3 7 1Q 3 Static CLEVR-Dialog[18] 7 7 2D 7 S”
History for Visual Dialog: Do we really need it? (2020, background): “Recently, there has been an increased interest in visual dialog, i.e. dialog-based interaction grounded in visual information (Chattopadhyay et al., 2017; De Vries et al., 2017; Seo et al., 2017; Guo et al., 2018; Shekhar et al., 2018; Kottur et al., 2019; Haber et al., 2019).”
VLN↻BERT: A Recurrent Vision-and-Language BERT for Navigation (2020, background): “However, we believe that it has a huge potential in addressing other tasks which require sequential interactions/decisions, such as language and visual dialog [5, 8, 21, 25, 45], dialog navigation [9, 40, 41] and action anticipation for reactive robot response [1, 13, 20].”
Learning Natural Language Generation with Truncated Reinforcement Learning (2022, methodology): “The external language model is trained on the mixture of CLOSURE (Bahdanau et al., 2019) and CLEVR-Dialog (Kottur et al., 2019) datasets.”
Revisiting Evaluation of Knowledge Base Completion Models (2020, methodology): “…of KGC through providing an appropriate and challenging dataset, this approach has been adopted as a common practice in other machine learning tasks such as visual reasoning [Johnson et al., 2017, Kottur et al., 2019], semantic parsing [Nangia and Bowman, 2018] and NLI [Zellers et al., 2018].”
Factorized Multimodal Transformer for Multimodal Sequential Learning (2019, background): “For example, consider multimodal language (face-to-face communication), where modalities of language, vision and acoustic are seamlessly used together for communicative intent (Kottur et al., 2019).”
Evaluating Very Long-Term Conversational Memory of LLM Agents (2024, background): “The image-grounded dialogue task is centered around responding to questions (Antol et al., 2015; Das et al., 2017; Kottur et al., 2019) or creating natural conversations related to specific images (Mostafazadeh et al., 2017; Shuster et al., 2018; Meng et al., 2020; Zheng et al., 2021).”
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, background): “2 CLEVR-Dialog The CLEVR-Dialog32 (Kottur et al., 2019) is developed for studying multi-round reasoning in visual dialog.”