Dense Procedure Captioning in Narrated Instructional Videos
By Botian Shi, Lei Ji, Yaobo Liang, Nan Duan, Peng Chen, Zhendong Niu, M. Zhou. 2019
59 citations recorded by the source corpus.
Understanding narrated instructional videos is important for both research and real-world web applications. Motivated by video dense captioning, we propose a model to generate procedure captions from narrated instructional videos which are a sequence of step-wise clips with description. Previous works on video dense captioning learn video segments and generate captions without considering transcripts. We argue that transcripts in narrated instructional videos can enhance video representation by providing fine-grained complimentary and semantic textual information. In this paper, we introduce a framework to (1) extract procedures by a cross-modality module, which fuses video content with the entire transcript; and (2) generate captions by encoding video frames as well as a snippet of transcripts within each extracted procedure. Experiments show that our model can achieve state-of-the-art performance in procedure extraction and captioning, and the ablation studies demonstrate that both the video frames and the transcripts are important for the task.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Cordelia Schmid
- Yejin Choi
- Mohit Bansal
- Ivan Laptev
- Jianfeng Gao
- Ali Farhadi
- Andrew Zisserman
- Josef Sivic
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Cordelia Schmid
- Andrew Zisserman
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Yi Yang
- Jitendra Malik
In Multimodal Learning
- Cordelia Schmid
- Arsha Nagrani
- Josef Sivic
- Ivan Laptev
- Antoine Miech
- Paul Hongsuck Seo
- Nan Duan
- Yejin Choi
Citations in the corpus
- MERLOT: Multimodal Neural Script Knowledge Models (2021, methodology): “Prior works have used web videos+ASR to build weakly-supervised object detectors [75], action detectors/classifiers [108, 5, 51, 72], instruction aligners [65, 4, 15], video captioners [85, 38, 74, 89], and visual reference resolvers [41].”
- UniViLM: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation (2020, methodology): “Shi et al. (2019) and Hessel et al. (2019) discuss the multimodal input with both video and transcript.”
- A Case Study on Combining ASR and Visual Features for Generating Instructional Video Captions (2019, background): “Palaskar et al. (2019) and Shi et al. (2019), contemporaneous with the submission of the present work, also examine ASR as a source of signal for generating how-to video captions.”
- Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning (2023, methodology): “[83] Botian Shi, Lei Ji, Yaobo Liang, Nan Duan, Peng Chen, Zhendong Niu, and Ming Zhou.”
- iPerceive: Applying Common-Sense Reasoning to Multi-Modal Dense Video Captioning and Video Question Answering (2020, background): “Hessel et al. [14] and Shi et al. [48] employ a Transformer architecture to encode both video frames and speech segments to generate captions for instructional cooking videos where speech and the captions are usually well-aligned with the visual content [38].”