A Case Study on Combining ASR and Visual Features for Generating Instructional Video Captions

By Jack Hessel, Bo Pang, Zhenhai Zhu, Radu Soricut. 2019

40 citations recorded by the source corpus.

Instructional videos get high-traffic on video sharing platforms, and prior work suggests that providing time-stamped, subtask annotations (e.g., “heat the oil in the pan”) improves user experiences. However, current automatic annotation methods based on visual features alone perform only slightly better than constant prediction. Taking cues from prior work, we show that we can improve performance significantly by considering automatic speech recognition (ASR) tokens as input. Furthermore, jointly modeling ASR tokens and visual features results in higher performance compared to training individually on either modality. We find that unstated background information is better explained by visual features, whereas fine-grained distinctions (e.g., “add oil” vs. “add olive oil”) are disambiguated more easily via ASR tokens.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Speech & Audio

In Signal Processing

Citations in the corpus

End-to-End Learning of Visual Representations From Uncurated Instructional Videos (2019, positive): “To avoid labeling visual data, several approaches have leveraged audio transcripts obtained from narrated videos using automatic speech recognition (ASR) as a way to supervise video models for object detection [4, 16, 56], captioning [34, 72], classification [3, 44, 49, 89], summarization [59] or re”
MERLOT RESERVE: Neural Script Knowledge through Vision and Language and Sound (2022, neutral): “, going beyond instructional videos [59]).”
MERLOT: Multimodal Neural Script Knowledge Models (2021, methodology): “Prior works have used web videos+ASR to build weakly-supervised object detectors [75], action detectors/classifiers [108, 5, 51, 72], instruction aligners [65, 4, 15], video captioners [85, 38, 74, 89], and visual reference resolvers [41].”
UniViLM: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation (2020, methodology): “…Shi et al., 2019; Palaskar et al., 2019; Hessel et al., 2019) which only use video signal, recent works (Shi et al., 2019; Palaskar et al., 2019; Hessel et al., 2019) study the multimodal captioning by taking both video and transcript as input, and show that incorporating transcript can largely…”
iPerceive: Applying Common-Sense Reasoning to Multi-Modal Dense Video Captioning and Video Question Answering (2020, background): “Hessel et al. [14] and Shi et al. [48] employ a Transformer architecture to encode both video frames and speech segments to generate captions for instructional cooking videos where speech and the captions are usually well-aligned with the visual content [38].”