GuessWhat?! Visual Object Discovery through Multi-modal Dialogue

By Harm de Vries, Florian Strub, A. P. Sarath Chandar, Olivier Pietquin, Hugo Larochelle, Aaron C. Courville. 2016

444 citations recorded by the source corpus.

We introduce GuessWhat?!, a two-player guessing game as a testbed for research on the interplay of computer vision and dialogue systems. The goal of the game is to locate an unknown object in a rich image scene by asking a sequence of questions. Higher-level image understanding, like spatial reasoning and language grounding, is required to solve the proposed task. Our key contribution is the collection of a large-scale dataset consisting of 150K human-played games with a total of 800K visual question-answer pairs on 66K images. We explain our design decisions in collecting the dataset and introduce the oracle and questioner tasks that are associated with the two players of the game. We prototyped deep learning models to establish initial baselines of the introduced tasks.

Subfields: Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes (2020, background): “[56] for a survey), with great progress on important problems such as visual question answering [3, 26] and image caption generation and retrieval [8, 88, 43, 68, 28], with offshoot tasks focusing specifically on visual reasoning [36], referring expressions [39], visual storytelling [57, 32], visual”
Large-scale Pretraining for Visual Dialog: A Simple State-of-the-Art Baseline (2019, methodology): “[1] introduced the task of Visual Dialog given an image, dialog history consisting of a sequence of question-”
Visual Dialog (2016, background): “Good data for this task should include dialogs that have (1) temporal continuity, (2) grounding in the image, and (3) mimic natural ‘conversational’ exchanges.”
Neural Approaches to Conversational AI (2018, neutral): “, Dhingra et al. (2017), Papangelis et al.”
Modulating early visual processing by language (2017, background): “We outline here the neural network architecture that was reported in the original GuessWhat?! paper [6].”
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization (2016, background): “Deep neural models based on Convolutional Neural Networks (CNNs) have enabled unprecedented breakthroughs in a variety of computer vision tasks, from image classification [33,24], object detection [21], semantic segmentation [37] to image captioning [55,7,18,29], visual question answering [3,20,42,4”
Learning Cooperative Visual Dialog Agents with Deep Reinforcement Learning (2017, background): “Note that the task assigned to subjects in [5] – “ask questions to imagine the hidden image better" – is similar to our game’s goal.”
The Hanabi Challenge: A New Frontier for AI Research (2019, background): “Algorithms have been developed to learn to solve riddles and referential games [6,73], gridworld games requiring coordination [74], object identification via question-and-answer dialog [75], and negotiation [7,8].”
Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Input (2018, background): “Joint modeling of images and natural language text has gained rapidly in popularity, encompassing tasks such as image captioning [3,4,5,6,7], visual question answering (VQA) [8,9,10,11,12], multimodal dialog [13], and text-to-image generation [14].”
Multi-step Reasoning via Recurrent Dual Attention for Visual Dialog (2019, background): “And [11] introduced the GuessWhat?! dataset, where the dialogs provided are more goal-oriented and aimed at object discovery within an image, through a series of yes/no questions between two dialog agents.”
Learning Visual Reasoning Without Strong Priors (2017, background): “These advances have in turn fueled research on the intersection of visual and linguistic learning [5, 6, 7, 8, 9].”
Attacking Visual Language Grounding with Adversarial Examples: A Case Study on Neural Image Captioning (2017, background): “…and Li, 2015; Vinyals et al., 2015; Donahue et al., 2015b; Pasunuru and Bansal, 2017; Venu-gopalan et al., 2015), visual dialog (Das et al., 2017; De Vries et al., 2017), visual question answering (Antol et al., 2015; Fukui et al., 2016; Lu et al., 2016; Zhu et al., 2017), visual story-telling…”
Guide Me: Interacting with Deep Networks (2018, background): “Other examples include generation of referring expressions [36, 44, 47, 62], segmentation or object retrieval from referring expressions [19, 29, 28], image captioning [16, 21, 33, 34, 48, 56] and visual dialog [18].”
End-to-end optimization of goal-driven and visually grounded dialogue systems (2017, methodology): “Ever since the formulation of the Turing Test, building systems that can meaningfully converse with humans has been a long-standing goal of Artificial Intelligence (AI).”
Visual Grounding in Video for Unsupervised Word Translation (2020, background): “Early work explored weak supervision and the correspondence problem between text annotations and image regions [7, 14], with more modern approaches exploring joint image-text word embeddings [17], or building a language conditioned attention map over the images in caption generation, visual question”
Multimodal Intelligence: Representation Learning, Information Fusion, and Applications (2019, background): “Furthermore, visual dialogue can be viewed as to answer a sequence of questions grounded in an image [232], [233], which extends VQA by requiring to generate more human like responses and to infer the context from the dialogue history.”
Towards Ecologically Valid Research on Language User Interfaces (2020, unknown): “One prominent example is the GuessWhat task (De Vries et al., 2017), a 20Q inspired game in which the user aims to ﬁnd a hidden object in an image.”
Two Causal Principles for Improving Visual Dialog (2019, background): “Another dataset GuessWhat?! proposed by [10] is a goal-driven visual dialog: questioner should locate an unknown object in a rich image scene by asking a”