Visual Dialog

By Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José M. F. Moura, Devi Parikh, Dhruv Batra. 2016

1,102 citations recorded by the source corpus.

We introduce the task of Visual Dialog, which requires an AI agent to hold a meaningful dialog with humans in natural, conversational language about visual content. Specifically, given an image, a dialog history, and a question about the image, the agent has to ground the question in image, infer context from history, and answer the question accurately. Visual Dialog is disentangled enough from a specific downstream task so as to serve as a general test of machine intelligence, while being grounded in vision enough to allow objective evaluation of individual responses and benchmark progress. We develop a novel two-person chat data-collection protocol to curate a large-scale Visual Dialog dataset (VisDial). VisDial contains 1 dialog (10 question-answer pairs) on ~140k images from the COCO dataset, with a total of ~1.4M dialog question-answer pairs. We introduce a family of neural encoder-decoder models for Visual Dialog with 3 encoders (Late Fusion, Hierarchical Recurrent Encoder and Memory Network) and 2 decoders (generative and discriminative), which outperform a number of sophisticated baselines. We propose a retrieval-based evaluation protocol for Visual Dialog where the AI agent is asked to sort a set of candidate answers and evaluated on metrics such as mean-reciprocal-rank of human response. We quantify gap between machine and human performance on the Visual Dialog task via human studies. Our dataset, code, and trained models will be released publicly at https://visualdialog.org. Putting it all together, we demonstrate the first visual chatbot!.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

CoQA: A Conversational Question Answering Challenge (2018, background): “Our work is closest to Das et al. (2017) and Saha et al.”
QuAC: Question Answering in Context (2018, background): “Most related to our effort is visual dialog (Das et al., 2017), which relies on images as evidence instead of text.”
Evaluating Large Language Models Trained on Code (2021, background): “…et al., 2020; Bao et al., 2021), audio and speech processing (Oord et al., 2016; 2018; Dhari-wal et al., 2020; Baevski et al., 2020), biology (Alley et al., 2019; Rives et al., 2021), and even across multiple modalities (Das et al., 2017; Lu et al., 2019; Ramesh et al., 2021; Zellers et al., 2021).”
Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions (2020, background): “, 2015), and more conversationally, visual dialogue (Das et al., 2017).”
ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks (2019, background): “3.2, we do miss some families of tasks including visually grounded dialog [4, 45], embodied tasks like question answering [7] and instruction following [8], and text generation tasks like image and video captioning [5].”
The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes (2020, background): “…88, 43, 68, 28], with offshoot tasks focusing speciﬁcally on visual reasoning [36], referring expressions [39], visual storytelling [57, 32], visual dialogue [9, 12], multimodal machine translation [18, 74], visual reasoning [75, 33, 71, 85, 27], visual common sense reasoning [91] and many others.”
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation (2022, background): “VisDial (Das et al., 2017) extends VQA in a natural conversational setting, where the model needs to predict an answer not only based on the image-question pair, but also considering the dialog history and the image’s caption.”
VD-BERT: A Unified Vision and Dialog Transformer with BERT (2020, methodology): “0 datasets 2 (Das et al., 2017).”
Large-scale Pretraining for Visual Dialog: A Simple State-of-the-Art Baseline (2019, methodology): “We use metrics introduced in [1].”
VizWiz Grand Challenge: Answering Visual Questions from Blind People (2018, neutral): “Our findings show the statistics found in the existing artificially constructed VQA datasets, nicely summarized in [13] and [20], match statistics observed in practice.”
Vision-and-Language Navigation: Interpreting Visually-Grounded Navigation Instructions in Real Environments (2017, background): “However, although many tasks combining visual and linguistic reasoning have been motivated by their potential robotic applications [4, 14, 21, 30, 42], none of these tasks allow an agent to move or control the camera.”
A Corpus for Reasoning about Natural Language Grounded in Photographs (2018, background): “Visual Dialog (Das et al., 2017) Dialogue Visual Question Answering 4 4”
Neural Approaches to Conversational AI (2018, neutral): “ the persona of the speaker or addressee (Li et al., 2016b; Al-Rfou et al., 2016), textual knowledge sources such as Foursquare (Ghazvininejad et al., 2018), the user’s or agent’s visual environment (Das et al., 2017a; Mostafazadeh et al., 2017), and affect or emotion of the user (Huber et al., 2018”
12-in-1: Multi-Task Vision and Language Representation Learning (2019, methodology): “ on multi-task learning for V&L tasks. Vision and language. While we address 12 V&L tasks in Sec.2.1, we do miss some families of tasks including image and video captioning [6], visual dialog [11], embodied question answering [10] and instruction following [3]. Different from earlier work [1”
TVQA: Localized, Compositional Video Question Answering (2018, methodology): “Alternative methods include sampling negative answers from other questions’ correct answers, either based on semantic similarity (Das et al., 2017; Jang et al., 2017) or randomly (Antol et al., 2015; Das et al., 2017).”
Modulating early visual processing by language (2017, background): “In particular, image captioning [16], visual question answering (VQA)[1, 23] and visually grounded dialogue systems[5, 6] constitute a popular set of example tasks for which large-scale datasets are now available.”
TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering (2017, background): “0 [12], and Visual Dialog [8], which all address image-based VQA.”
Inferring and Executing Programs for Visual Reasoning (2017, background): “1Memory is likely indispensable in more complex settings such as visual dialogues or SHRDLU [6, 42].”
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization (2016, background): “Deep neural models based on Convolutional Neural Networks (CNNs) have enabled unprecedented breakthroughs in a variety of computer vision tasks, from image classification [33,24], object detection [21], semantic segmentation [37] to image captioning [55,7,18,29], visual question answering [3,20,42,4”