End-to-end Audio Visual Scene-aware Dialog Using Multimodal Attention-based Video Features
By Chiori Hori, Huda AlAmri, Jue Wang, Gordon Wichern, Takaaki Hori, Anoop Cherian, Tim K. Marks, Vincent Cartillier, Raphael Gontijo Lopes, Abhishek Das and 3 more. 2018
132 citations recorded by the source corpus.
In order for machines interacting with the real world to have conversations with users about the objects and events around them, they need to understand dynamic audiovisual scenes. The recent revolution of neural network models allows us to combine various modules into a single end-to-end differentiable network. As a result, Audio Visual Scene-Aware Dialog (AVSD) systems for real-world applications can be developed by integrating state-of-the-art technologies from multiple research areas, including end-to-end dialog technologies, visual question answering (VQA) technologies, and video description technologies. In this paper, we introduce a new data set of dialogs about videos of human behaviors, as well as an end-to-end Audio Visual Scene-Aware Dialog (AVSD) model, trained using this new data set, that generates responses in a dialog about a video. By using features that were developed for multimodal attention-based video description, our system improves the quality of generated dialog about dynamic video scenes.
Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Dialogue & Conversational AI
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Cordelia Schmid
- Dacheng Tao
- Trevor Darrell
- Andrew Zisserman
- Xiaogang Wang
- Ming-Hsuan Yang
- Jitendra Malik
In Multimodal Learning
- Cordelia Schmid
- Arsha Nagrani
- Paul Hongsuck Seo
- Steven C. H. Hoi
- Xudong Lin
- Dietrich Klakow
- Marimuthu Kalimuthu
- Aditya Mogadala
In Dialogue & Conversational AI
Citations in the corpus
- Look Before you Speak: Visually Contextualized Utterances (2020, background): “A number of works show promising results on this task [14, 32, 42, 45, 48].”
- VX2TEXT: End-to-End Learning of Video-Based Text Generation From Multimodal Inputs (2021, background): “Several approaches have been introduced for these tasks [34, 3, 23, 42, 16, 41, 25, 52, 20, 27, 30, 55, 17, 31].”
- Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, methodology): “Model B-1 B-2 B-3 B-4 METEOR CIDEr Att-base (Hori et al., 2019) 0.”
- Multimodal Transformer Networks for End-to-End Video-Grounded Dialogue Systems (2019, methodology): “Figure 5 summarizes the CIDEr measures of the responses generated by our Base model and the baseline (Hori et al., 2018) by their position in dialogue e.g. 1st...10th turn.”
- AudioCaps: Generating Captions for Audios in The Wild (2019, background): “Recently, there have been some efforts to solve video captioning with audio input (Hori et al., 2017, 2018; Wang et al., 2018).”