Audio Visual Scene-Aware Dialog

By Huda AlAmri, Vincent Cartillier, Abhishek Das, Jue Wang, Stefan Lee, Peter Anderson, Irfan Essa, Devi Parikh, Dhruv Batra, Anoop Cherian and 2 more. 2019

230 citations recorded by the source corpus.

We introduce the task of scene-aware dialog. Our goal is to generate a complete and natural response to a question about a scene, given video and audio of the scene and the history of previous turns in the dialog. To answer successfully, agents must ground concepts from the question in the video while leveraging contextual cues from the dialog history. To benchmark this task, we introduce the Audio Visual Scene-Aware Dialog (AVSD) Dataset. For each of more than 11,000 videos of human actions from the Charades dataset, our dataset contains a dialog about the video, plus a final summary of the video by one of the dialog participants. We train several baseline systems for this task and evaluate the performance of the trained models using both qualitative and quantitative metrics. Our results indicate that models must utilize all the available inputs (video, audio, question, and dialog history) to perform best on this dataset.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Dialogue & Conversational AI

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Cordelia Schmid
  4. Trevor Darrell
  5. Andrew Zisserman
  6. Yi Yang
  7. Xiaogang Wang
  8. Ming-Hsuan Yang

In Multimodal Learning

  1. Cordelia Schmid
  2. Arsha Nagrani
  3. Paul Hongsuck Seo
  4. Nan Duan
  5. Devi Parikh
  6. Xiaodan Liang
  7. Lorenzo Torresani
  8. Xudong Lin

In Dialogue & Conversational AI

  1. Arsha Nagrani
  2. Cordelia Schmid
  3. Paul Hongsuck Seo
  4. Marimuthu Kalimuthu
  5. Dietrich Klakow
  6. Aditya Mogadala
  7. Can Xu
  8. Daxin Jiang

Citations in the corpus