Multimodal Abstractive Summarization for How2 Videos
By Shruti Palaskar, Jindřich Libovický, Spandana Gella, Florian Metze. 2019
110 citations recorded by the source corpus.
In this paper, we study abstractive summarization for open-domain videos. Unlike the traditional text news summarization, the goal is less to “compress” text information but rather to provide a fluent textual summary of information that has been collected and fused from different source modalities, in our case video and audio transcripts (or text). We show how a multi-source sequence-to-sequence model with hierarchical attention can integrate information from different modalities into a coherent output, compare various models trained with different modalities and present pilot experiments on the How2 corpus of instructional videos. We also propose a new evaluation metric (Content F1) for abstractive summarization task that measures semantic adequacy rather than fluency of the summaries, which is covered by metrics like ROUGE and BLEU.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Summarization
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yejin Choi
- Mohit Bansal
- Jianfeng Gao
- Pascale Fung
- Luke S. Zettlemoyer
- Rowan Zellers
- Ali Farhadi
- Zhe Gan
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Cordelia Schmid
- Xiaogang Wang
- Yejin Choi
- Ming-Hsuan Yang
- Yi Yang
In Multimodal Learning
In Summarization
Citations in the corpus
- MERLOT: Multimodal Neural Script Knowledge Models (2021, methodology): “Prior works have used web videos with ASR to build weakly-supervised object detectors [87], action detectors/classifiers [120, 6, 62, 84], instruction aligners [77, 5, 19], video captioners [96, 46, 86, 101], and visual reference resolvers [49].”
- UniViLM: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation (2020, methodology): “Different from existing works (Sun et al., 2019b,a; Krishna et al., 2017; Zhou et al., 2018a,b; Shi et al., 2019; Palaskar et al., 2019; Hessel et al., 2019) which only use video signal, recent works (Shi et al., 2019; Palaskar et al., 2019; Hessel et al., 2019) study the multimodal captioning by…”
- A Case Study on Combining ASR and Visual Features for Generating Instructional Video Captions (2019, background): “Palaskar et al. (2019) and Shi et al. (2019), contemporaneous with the submission of the present work, also examine ASR as a source of signal for generating how-to video captions.”
- Video Summarization Using Deep Neural Networks: A Survey (2021, background): “Addressing the video summarization task from a slightly different perspective, other multimodal algorithms generate a text-based summary of the video [69], [70], [71], and extend this output by extracting one representative key-frame [72].”
- Multi-document Summarization via Deep Learning Techniques: A Survey (2020, background): “This kind of multi-modal summary can improve the satisfaction of users [158], (ii) text + video: Based on the video and its subtitles, generating a concise text summary that describes the main context of video [97].”
- Vision Guided Generative Pre-trained Language Models for Multimodal Abstractive Summarization (2021, methodology): “Multimodal abstractive summarization (MAS) aims to take advantage of data from multiple modalities and provides a short, concise and readable textual summary to let users quickly acquire their essential information (Sanabria et al., 2018; Palaskar et al., 2019; Liu et al., 2020).”
- A Metaverse: Taxonomy, Components, Applications, and Open Challenges (2022, background): “Text and images in social media posts do not have the same meaning but instead have more complex meanings that intersect semantically [162]–[164].”