Two Can Play This Game: Visual Dialog with Discriminative Question Generation and Answering

By Unnat Jain, Svetlana Lazebnik, Alexander Gerhard Schwing. 2018

84 citations recorded by the source corpus.

Human conversation is a complex mechanism with subtle nuances. It is hence an ambitious goal to develop artificial intelligence agents that can participate fluently in a conversation. While we are still far from achieving this goal, recent progress in visual question answering, image captioning, and visual question generation shows that dialog systems may be realizable in the not too distant future. To this end, a novel dataset was introduced recently and encouraging results were demonstrated, particularly for question answering. In this paper, we demonstrate a simple symmetric discriminative baseline, that can be applied to both predicting an answer as well as predicting a question. We show that this method performs on par with the state of the art, even memory net based methods. In addition, for the first time on the visual dialog dataset, we assess the performance of a system asking questions, and demonstrate how visual dialog can be generated from discriminative question generation and question answering.

Subfields: Multimodal Learning, Dialogue & Conversational AI

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Multimodal Learning

In Dialogue & Conversational AI

Citations in the corpus

Large-scale Pretraining for Visual Dialog: A Simple State-of-the-Art Baseline (2019, background): “Recent years have seen incredible progress in Visual Dialog [1, 5–25], spurred in part by the initial efforts of Das et al.”
Multi-step Reasoning via Recurrent Dual Attention for Visual Dialog (2019, background): “In addition to answering questions, the ability of generating sequence of questions is also investigated in [22, 33].”
Efficient Attention Mechanism for Handling All the Interactions between Many Inputs with Application to Visual Dialog (2019, methodology): “0 datasets, which include LF, HRE, MN [8], LF-Att, MN-Att (with attention) [8], SAN [42], AMEM [35], SF [18], HCIAE [27] and Sequential CoAttention model (CoAtt) [40], Synergistic [15], FGA [34], GNN [48], RvA [30], CorefNMN [22], HCAN [41], DAN [19], and ReDAN [13].”
Audio Visual Scene-Aware Dialog (2019, background): “In order to combine the features from these diverse inputs, we follow recent work in visually grounded dialog [10]: simply concatenate the features, and allow fusion to occur through fully-connected layers.”
Making History Matter: Gold-Critic Sequence Training for Visual Dialog (2019, background): “…history question-answer sentences, the image, and the current question sentence, into a vector representation; then, the decoder fuses those vectors ( e.g ., usually by concatenation) and decode the fused one into answer sentences ( e.g ., by generation [7, 21, 31] or candidate ranking [16, 18]).”
Straight to the Facts: Learning Knowledge Base Retrieval for Factual Visual Question Answering (2018, methodology): “Most of the methods developed for visual question answering [2,8,10,18,19,20,21,22,23,24,12,27,29,30,31,6,7,32,33,34] focus exclusively on answering questions related to observed content.”
Large-Scale Answerer in Questioner's Mind for Visual Dialog Question Generation (2019, methodology): “The performance of AQM+ can be boosted further by employing the models recently proposed in the visual dialog field such as other question generator models [11] and question answering models [17].”
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, background): “, 2017) VGG LSTM AMEM ADAM 7 (Jain et al., 2018) VGG LSTM SF ADAM 7 (Kottur et al.”
Two Causal Principles for Improving Visual Dialog (2019, methodology): “All of the existing approaches in the VisDial task are based on the typical encoder-decoder framework [15, 12, 33, 11, 32, 48].”
Reasoning Visual Dialogs With Structural and Partial Observations (2019, methodology): “The goal is to answer questions posed in natural language about images [9], or recover a follow-up question based on the dialog history [22].”
Text Augmentation Using BERT for Image Captioning (2020, background): “The datasets [22,23] for this task contain only a limited number of replicas.”
Convolutional Image Captioning (2017, background): “LSTM nets have been considered as the de-facto standard for vision-language tasks of image captioning [5, 16, 37, 39, 38], visual question answering [3, 30, 28], question generation [14, 20], and visual dialog [7, 13], due to their compelling ability to memorize long-term depen-”