Are You Talking to Me? Reasoned Visual Dialog Generation Through Adversarial Learning

By Qi Wu, Peng Wang, Chunhua Shen, Ian David Reid, Anton van den Hengel. 2017

133 citations recorded by the source corpus.

The visual dialog task requires an agent to engage in a conversation about an image with a human. It represents an extension of the visual question answering task in that the agent needs to answer a question about an image, but it needs to do so in light of the previous dialog that has taken place. The key challenge in visual dialog is thus maintaining a consistent, and natural dialog while continuing to answer questions correctly. We present a novel approach that combines Reinforcement Learning and Generative Adversarial Networks (GANS) to generate more human-like responses to questions. The GAN helps overcome the relative paucity of training data, and the tendency of the typical MLE-based approach to generate overly terse answers. Critically, the GAN is tightly integrated into the attention mechanism that generates human-interpretable reasons for each answer. This means that the discriminative model of the GAN has the task of assessing whether a candidate answer is generated by a human or not, given the provided reason. This is significant because it drives the generative model to produce high quality answers that are well supported by the associated reasoning. The method also generates the state-of-the-art results on the primary benchmark.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Dialogue & Conversational AI

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Ming-Hsuan Yang
  4. Trevor Darrell
  5. Xiaogang Wang
  6. Cordelia Schmid
  7. Yi Yang
  8. Jitendra Malik

In Multimodal Learning

  1. Zhe Gan
  2. Linjie Li
  3. Jingjing Liu
  4. Jianfeng Gao
  5. Mohit Bansal
  6. Devi Parikh
  7. Yu Cheng
  8. Hanwang Zhang

In Dialogue & Conversational AI

  1. Hanwang Zhang
  2. Takayuki Okatani
  3. Van-Quang Nguyen
  4. Masanori Suganuma
  5. Can Xu
  6. Daxin Jiang
  7. Xiuyi Chen
  8. Zhengjun Zha

Citations in the corpus