Speech2Action: Cross-Modal Supervision for Action Recognition

By Arsha Nagrani, Chen Sun, David A. Ross, Rahul Sukthankar, Cordelia Schmid, Andrew Zisserman. 2020

62 citations recorded by the source corpus.

Is it possible to guess human action from dialogue alone? In this work we investigate the link between spoken words and actions in movies. We note that movie screenplays describe actions, as well as contain the speech of characters and hence can be used to learn this correlation with no additional supervision. We train a BERT-based Speech2Action classifier on over a thousand movie screenplays, to predict action labels from transcribed speech segments. We then apply this model to the speech segments of a large unlabelled movie corpus (188M speech segments from 288K movies). Using the predictions of this model, we obtain weak action labels for over 800K video clips. By training on these video clips, we demonstrate superior action recognition performance on standard action recognition benchmarks, without using a single manually labelled action example.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Learning Video Representations from Textual Web Supervision (2020, background): “Prior works have leveraged ambient sound [52, 5, 4, 71, 34, 51, 55], dialogue [49], and narration [70, 2, 72, 73, 46, 45, 3], all of which of which serve as useful signals.”
Look Before you Speak: Visually Contextualized Utterances (2020, background): “There have also been attempts to use transcribed speech in videos as a source of weak supervision [52, 60, 76, 77], where the goal is to learn a good visual encoder, and consequently such works are largely evaluated only on downstream tasks that involve unimodal video frame inputs.”
Condensed Movies: Story Based Retrieval with Contextual Embeddings (2020, background): “Early works, however, focused on using film and TV to learn human identity [21,22,23,24,25,26] or human actions [27,28,29,30,31] from the scripts or captions accompanying movies.”
Human Action Recognition From Various Data Modalities: A Review (2020, background): “[353] introduced the speech modality, neither visual nor sensor modalities,”
Multimodal Learning With Transformers: A Survey (2022, unknown): “multispectral image [15], video [7], audio/speech/music [14], [16], [17], table [18], scene graph/layout [19], [20], [21], [22], pose skeleton [23], SQL [24], [25], recipe [26], programming”
Multi-modal Self-Supervision from Generalized Data Transformations (2020, methodology): “ce [5, 6, 9, 71, 78] and synchronization [20, 55, 77] between the audio and visual streams. A number of recent papers have leveraged speech as a weak supervisory signal to train video representations [62, 68, 72, 96, 97]. Other works incorporate optical ﬂow and other modalities [64, 83, 98, 114] to ”
Balanced Multimodal Learning via On-the-fly Gradient Modulation (2022, background): “, action recognition [10, 23, 29], audiovisual speech recognition [16, 33], and visual question answering [2, 19].”