Audio Visual Scene-Aware Dialog
By Huda AlAmri, Vincent Cartillier, Abhishek Das, Jue Wang, Stefan Lee, Peter Anderson, Irfan Essa, Devi Parikh, Dhruv Batra, Anoop Cherian and 2 more. 2019
230 citations recorded by the source corpus.
We introduce the task of scene-aware dialog. Our goal is to generate a complete and natural response to a question about a scene, given video and audio of the scene and the history of previous turns in the dialog. To answer successfully, agents must ground concepts from the question in the video while leveraging contextual cues from the dialog history. To benchmark this task, we introduce the Audio Visual Scene-Aware Dialog (AVSD) Dataset. For each of more than 11,000 videos of human actions from the Charades dataset, our dataset contains a dialog about the video, plus a final summary of the video by one of the dialog participants. We train several baseline systems for this task and evaluate the performance of the trained models using both qualitative and quantitative metrics. Our results indicate that models must utilize all the available inputs (video, audio, question, and dialog history) to perform best on this dataset.
Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Dialogue & Conversational AI
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Cordelia Schmid
- Trevor Darrell
- Andrew Zisserman
- Yi Yang
- Xiaogang Wang
- Ming-Hsuan Yang
In Multimodal Learning
- Cordelia Schmid
- Arsha Nagrani
- Paul Hongsuck Seo
- Nan Duan
- Devi Parikh
- Xiaodan Liang
- Lorenzo Torresani
- Xudong Lin
In Dialogue & Conversational AI
- Arsha Nagrani
- Cordelia Schmid
- Paul Hongsuck Seo
- Marimuthu Kalimuthu
- Dietrich Klakow
- Aditya Mogadala
- Can Xu
- Daxin Jiang
Citations in the corpus
- Large-scale Pretraining for Visual Dialog: A Simple State-of-the-Art Baseline (2019, neutral): “video understanding [26], navigation assistants [27–29].”
- Look Before you Speak: Visually Contextualized Utterances (2020, background): “Related to our work is the task of scene-aware dialog prediction [2, 33], where the goal is to answer questions grounded to a video clip input, given a manually created dialog history.”
- PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable (2019, background): “Table 8 provides the generated responses on DSTC7-AVSD, where two participants discuss the objects and events in a video.”
- VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs (2024, background): “For AVSSD and AVSD, we employ a GPT-assisted evaluation as the same as the protocols in OE-VQA.”
- M5Product: A Multi-modal Pretraining Benchmark for E-commercial Product Downstream Tasks (2021, background): “LJ Speech [15] and SQuAD [16] are the classical audio/text datasets for the voice synthesis and audio Q & A. Similarly, most video/text datasets [17, 18, 19, 20, 21, 22, 23, 24] are mainly for video Q & A with limited number of samples.”
- ProphetNet-X: Large-Scale Pre-training Models for English, Chinese, Multi-lingual, Dialog, and Code Generation (2021, methodology): “ProphetNet-X achieves new state-of-the-art results on 10 benchmarks, including Chinese summarization (MATINF-SUMM (Xu et al., 2020a) and LC-STS (Hu et al., 2015)), Chinese question answering (MATINF-QA (Xu et al., 2020a)), cross-lingual generation (XGLUE NTG (Liang et al., 2020) and XGLUE QG (Liang ”
- On Modality Bias in the TVQA Dataset (2020, unknown): “The AVSD dataset [3] is a visual dialog dataset i.e. multiple rounds of ongoing QA ‘dialog’ about a given video.”
- Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, background): “The aim of video dialog is to leverage scene information containing both audio (can be transcribed as subtitles) and visual frames to hold a dialog with humans in natural language about the content (Alamri et al., 2019b, 2019a).”
- VX2TEXT: End-to-End Learning of Video-Based Text Generation From Multimodal Inputs (2021, background): “Prime examples include datasets for image or video captioning [10, 38, 51, 24, 56, 28], question answering (QA) [5, 13, 54, 58, 19, 36, 46, 26], as well as audio-visual dialog [11, 1].”
- What Makes Training Multi-Modal Classification Networks Hard? (2019, background): “Anecdotally, the performance drop with multiple input streams appears to be common and was noted in [24, 3, 38, 44].”