FLIPDIAL: A Generative Model for Two-Way Visual Dialogue

By Daniela Massiceti, N. Siddharth, Puneet Kumar Dokania, Philip H. S. Torr. 2018

42 citations recorded by the source corpus.

We present FLIPDIAL, a generative model for Visual Dialogue that simultaneously plays the role of both participants in a visually-grounded dialogue. Given context in the form of an image and an associated caption summarising the contents of the image, FLIPDIAL learns both to answer questions and put forward questions, capable of generating entire sequences of dialogue (question-answer pairs) which are diverse and relevant to the image. To do this, FLIPDIAL relies on a simple but surprisingly powerful idea: it uses convolutional neural networks (CNNs) to encode entire dialogues directly, implicitly capturing dialogue context, and conditional VAEs to learn the generative model, FLIPDIAL outperforms the state-of-the-art model in the sequential answering task (1VD) on the VisDial dataset by 5 points in Mean Rank using the generated answers. We are the first to extend this paradigm to full two-way visual dialogue (2VD), where our model is capable of generating both questions and answers in sequence based on a visual input, for which we propose a set of novel evaluation measures and metrics.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Dialogue & Conversational AI

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Large-scale Pretraining for Visual Dialog: A Simple State-of-the-Art Baseline (2019, background): “Follow-up works on visual dialog have explored the use of deep reinforcement learning [3, 4, 17], knowledge transfer from discriminative to generative decoders [5], conditional variational autoencoders [6], generative adversarial networks [7], attention mechanisms for visual coreference resolution [”
A Survey of Reinforcement Learning Informed by Natural Language (2019, background): “Recent work on visual Q&A (VQA) have produced a first exploration of multi-modal settings in which agents are tasked with performing both visual and languagebased reasoning [Antol et al., 2015; Johnson et al., 2017; Massiceti et al., 2018].”
Improving Generative Visual Dialog by Answering Diverse Questions (2019, unknown): “Closest to our setting is work on using conditional variational autoencoders for self-talk in visual dialog (Massiceti et al., 2018), where diversity is not explicitly modeled but is measured via metrics speciﬁc to the proposed architecture.”
Reasoning Visual Dialogs With Structural and Partial Observations (2019, unknown): “sequences of dialog [41], unifying visual question generation and visual question answering in a dual learning framework [22], combining RL and Generative Adversarial Networks (GANs) to generate more human-like answers [62].”
Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Models (2019, methodology): “For the language data, we employ a CNN encoder and decoder following Kalchbrenner et al. (2014); Massiceti et al. (2018b); Pham et al. (2016), learning an embedding for words in the process.”