End-to-end Audio Visual Scene-aware Dialog Using Multimodal Attention-based Video Features

By Chiori Hori, Huda AlAmri, Jue Wang, Gordon Wichern, Takaaki Hori, Anoop Cherian, Tim K. Marks, Vincent Cartillier, Raphael Gontijo Lopes, Abhishek Das and 3 more. 2018

132 citations recorded by the source corpus.

In order for machines interacting with the real world to have conversations with users about the objects and events around them, they need to understand dynamic audiovisual scenes. The recent revolution of neural network models allows us to combine various modules into a single end-to-end differentiable network. As a result, Audio Visual Scene-Aware Dialog (AVSD) systems for real-world applications can be developed by integrating state-of-the-art technologies from multiple research areas, including end-to-end dialog technologies, visual question answering (VQA) technologies, and video description technologies. In this paper, we introduce a new data set of dialogs about videos of human behaviors, as well as an end-to-end Audio Visual Scene-Aware Dialog (AVSD) model, trained using this new data set, that generates responses in a dialog about a video. By using features that were developed for multimodal attention-based video description, our system improves the quality of generated dialog about dynamic video scenes.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Dialogue & Conversational AI

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Cordelia Schmid
  3. Dacheng Tao
  4. Trevor Darrell
  5. Andrew Zisserman
  6. Xiaogang Wang
  7. Ming-Hsuan Yang
  8. Jitendra Malik

In Multimodal Learning

  1. Cordelia Schmid
  2. Arsha Nagrani
  3. Paul Hongsuck Seo
  4. Steven C. H. Hoi
  5. Xudong Lin
  6. Dietrich Klakow
  7. Marimuthu Kalimuthu
  8. Aditya Mogadala

In Dialogue & Conversational AI

  1. Arsha Nagrani
  2. Cordelia Schmid
  3. Paul Hongsuck Seo
  4. Marimuthu Kalimuthu
  5. Dietrich Klakow
  6. Aditya Mogadala

Citations in the corpus