Cross-Modal Supervision for Learning Active Speaker Detection in Video

By Punarjay Chakravarty, Tinne Tuytelaars. 2016

78 citations recorded by the source corpus.

In this paper, we show how to use audio to supervise the learning of active speaker detection in video. Voice Activity Detection (VAD) guides the learning of the vision-based classifier in a weakly supervised manner. The classifier uses spatio-temporal features to encode upper body motion - facial expressions and gesticulations associated with speaking. We further improve a generic model for active speaker detection by learning person specific models. Finally, we demonstrate the online adaptation of generic models learnt on one dataset, to previously unseen people in a new dataset, again using audio (VAD) for weak supervision. The use of temporal continuity overcomes the lack of clean training data. We are the first to present an active speaker detection system that learns on one audio-visual dataset and automatically adapts to speakers in a new dataset. This work can be seen as an example of how the availability of multi-modal data allows us to learn a model without the need for supervision, by transferring knowledge from one modality to another.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

VoxCeleb: A Large-Scale Speaker Identification Dataset (2017, background): “Precision-recall curves for both tasks on their respective benchmark datasets [30, 31] are shown in Figure 1 right, and the values at the operating point are given in Table 3.”
Ava Active Speaker: An Audio-Visual Dataset for Active Speaker Detection (2019, background): “The closest re-lated work is that of Chakravarty et al. [5, 6, 7], and their entire experimental data includes seven 30 -min PhD thesis presentations and segments of a single video with a panel discussion from YouTube.”
Voxceleb: Large-scale speaker verification in the wild (2020, methodology): “Precision-recall curves for both tasks on their respective benchmark datasets (Parkhi et al., 2015; Chakravarty and Tuytelaars, 2016) are shown in Fig.”
Out of Time: Automated Lip Sync in the Wild (2016, methodology): “The results of [4] have been digitised from Figure 3b of their paper, and are accurate to around ±0.”