Multimodal Transformer Networks for End-to-End Video-Grounded Dialogue Systems

By Hung Le, Doyen Sahoo, Nancy F. Chen, Steven C. H. Hoi. 2019

123 citations recorded by the source corpus.

Developing Video-Grounded Dialogue Systems (VGDS), where a dialogue is conducted based on visual and audio aspects of a given video, is significantly more challenging than traditional image or text-grounded dialogue systems because (1) feature space of videos span across multiple picture frames, making it difficult to obtain semantic information; and (2) a dialogue agent must perceive and process information from different modalities (audio, video, caption, etc.) to obtain a comprehensive understanding. Most existing work is based on RNNs and sequence-to-sequence architectures, which are not very effective for capturing complex long-term dependencies (like in videos). To overcome this, we propose Multimodal Transformer Networks (MTN) to encode videos and incorporate information from different modalities. We also propose query-aware attention through an auto-encoder to extract query-aware features from non-text modalities. We develop a training procedure to simulate token-level decoding to improve the quality of generated responses during inference. We get state of the art performance on Dialogue System Technology Challenge 7 (DSTC7). Our model also generalizes to another multimodal visual-grounded dialogue task, and obtains promising performance.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Dialogue & Conversational AI

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Christoph Feichtenhofer
  2. Lorenzo Torresani
  3. Trevor Darrell
  4. Mohit Bansal
  5. Yanghao Li
  6. Kristen Grauman
  7. Jitendra Malik
  8. Cordelia Schmid

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Jitendra Malik
  3. Dacheng Tao
  4. Trevor Darrell
  5. Xiaogang Wang
  6. Ming-Hsuan Yang
  7. Cordelia Schmid
  8. Andrew Zisserman

In Multimodal Learning

  1. Shih-Fu Chang
  2. Christoph Feichtenhofer
  3. Lorenzo Torresani
  4. Haoqi Fan
  5. Xudong Lin
  6. Erik Cambria
  7. Bo Xiong
  8. Tom Young

Citations in the corpus